[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:12 dist-test-slave-k0b7 krb5kdc[3166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:12 dist-test-slave-k0b7 krb5kdc[3166](info): set up 2 sockets
Sep 29 01:00:12 dist-test-slave-k0b7 krb5kdc[3166](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 01:00:13.071805 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:13 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:13.281014 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:45099
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:13.657166 3179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:13.657630 3179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:13.657793 3179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:13.666841 3179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:13.666985 3179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:13.667089 3179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:13.667169 3179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:13.667235 3179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:13.677889 3179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:45099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:13.679451 3179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:13.680939 3179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:13.695624 3185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:13.695680 3187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:13.695680 3184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:13.696604 3179 server_base.cc:1034] running on GCE node
I20260929 01:00:13.697530 3179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:13.699105 3179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:13.700403 3179 hybrid_clock.cc:648] HybridClock initialized: now 1790643613700329 us; error 66 us; skew 500 ppm
I20260929 01:00:13.700793 3179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:13 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643613, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:13.709585 3179 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:13.711818 3179 webserver.cc:460] Webserver started at http://127.3.21.254:46295/ using document root <none> and password file <none>
I20260929 01:00:13.712576 3179 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:13.712683 3179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:13.712942 3179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:13.716269 3179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "b43431b444e646e7980f4b40339fb6ec"
format_stamp: "Formatted at 2026-09-29 01:00:13 on dist-test-slave-k0b7"
server_key: "ffd5fb26f8c0349caf014f81824f9153"
server_key_iv: "d7b2f5d5fd9d82856846f105539c64df"
server_key_version: "encryptionkey@0"
I20260929 01:00:13.717061 3179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "b43431b444e646e7980f4b40339fb6ec"
format_stamp: "Formatted at 2026-09-29 01:00:13 on dist-test-slave-k0b7"
server_key: "ffd5fb26f8c0349caf014f81824f9153"
server_key_iv: "d7b2f5d5fd9d82856846f105539c64df"
server_key_version: "encryptionkey@0"
I20260929 01:00:13.723285 3179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:00:13.727825 3194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:13.729657 3179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:13.729840 3179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "b43431b444e646e7980f4b40339fb6ec"
format_stamp: "Formatted at 2026-09-29 01:00:13 on dist-test-slave-k0b7"
server_key: "ffd5fb26f8c0349caf014f81824f9153"
server_key_iv: "d7b2f5d5fd9d82856846f105539c64df"
server_key_version: "encryptionkey@0"
I20260929 01:00:13.730052 3179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:13.762202 3179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:13.767444 3179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:13.767699 3179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:13.790659 3179 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:45099
I20260929 01:00:13.790736 3245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:45099 every 8 connection(s)
I20260929 01:00:13.792321 3179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:13.793902 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3179
I20260929 01:00:13.794240 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:13.796428 3159 external_mini_cluster.cc:1428] Setting key d5ffd10cd2ea1eb6852b65aba865bb79
I20260929 01:00:13.797019 3246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:13.809453 3246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec: Bootstrap starting.
I20260929 01:00:13.814090 3246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:13.815855 3246 log.cc:824] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:13.820086 3246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec: No bootstrap required, opened a new log
I20260929 01:00:13.830619 3246 raft_consensus.cc:348] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43431b444e646e7980f4b40339fb6ec" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45099 } }
I20260929 01:00:13.831058 3246 raft_consensus.cc:374] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:13.831197 3246 raft_consensus.cc:729] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43431b444e646e7980f4b40339fb6ec, State: Initialized, Role: FOLLOWER
I20260929 01:00:13.832054 3246 consensus_queue.cc:260] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43431b444e646e7980f4b40339fb6ec" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45099 } }
I20260929 01:00:13.832371 3246 raft_consensus.cc:388] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:13.832569 3246 raft_consensus.cc:482] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:13.832762 3246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 0 FOLLOWER]: Advancing to term 1
Sep 29 01:00:13 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:13.836028 3246 raft_consensus.cc:504] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43431b444e646e7980f4b40339fb6ec" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45099 } }
I20260929 01:00:13.836768 3246 leader_election.cc:302] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b43431b444e646e7980f4b40339fb6ec; no voters:
I20260929 01:00:13.837538 3246 leader_election.cc:288] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:13.837720 3251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:13.838922 3251 raft_consensus.cc:686] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [term 1 LEADER]: Becoming Leader. State: Replica: b43431b444e646e7980f4b40339fb6ec, State: Running, Role: LEADER
I20260929 01:00:13.839512 3251 consensus_queue.cc:237] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43431b444e646e7980f4b40339fb6ec" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45099 } }
I20260929 01:00:13.840206 3246 sys_catalog.cc:564] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:13.842412 3253 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b43431b444e646e7980f4b40339fb6ec. Latest consensus state: current_term: 1 leader_uuid: "b43431b444e646e7980f4b40339fb6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43431b444e646e7980f4b40339fb6ec" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45099 } } }
I20260929 01:00:13.843322 3253 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:13.843910 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b43431b444e646e7980f4b40339fb6ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43431b444e646e7980f4b40339fb6ec" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45099 } } }
I20260929 01:00:13.844256 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:13.844332 3258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:13.853411 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:13.801149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.3.21.254:45099)
0929 01:00:13.802231 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:13.802249 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:13.803804 (+ 1555us) server_negotiation.cc:407] Connection header received
0929 01:00:13.815846 (+ 12042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:13.815887 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:13.817068 (+ 1181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:13.817612 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:13.821319 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:13.823906 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:13.825999 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:13.827400 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:13.827638 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:13.836072 (+ 8434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:13.836086 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:13.836109 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:13.836180 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:13.841161 (+ 4981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:13.842704 (+ 1543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:13.842715 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:13.842727 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:13.843411 (+ 684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:13.844312 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:13.844365 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:13.844412 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:13.845604 (+ 1192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:13.849856 (+ 4252us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:13.850302 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 01:00:13.851190 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":463,"threads_started":1}
I20260929 01:00:13.861913 3258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:13.875912 3258 catalog_manager.cc:1347] Generated new cluster ID: 818f6a664e1342aebaba9f4bf9b4e7f3
I20260929 01:00:13.876061 3258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:13.895812 3258 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:13.898118 3258 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:13.918815 3258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b43431b444e646e7980f4b40339fb6ec: Generated new TSK 0
I20260929 01:00:13.919862 3258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:13.977694 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45099
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:14.391844 3272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:14.392220 3272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:14.392397 3272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:14.402118 3272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:14.402324 3272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:14.402596 3272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:14.415010 3272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:14.416803 3272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:14.418567 3272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:14.438427 3278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:14.438599 3277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:14.438760 3272 server_base.cc:1034] running on GCE node
W20260929 01:00:14.438580 3280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:14.439972 3272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:14.441237 3272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:14.442632 3272 hybrid_clock.cc:648] HybridClock initialized: now 1790643614442551 us; error 73 us; skew 500 ppm
I20260929 01:00:14.443049 3272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:14 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:14.447590 3272 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:14.449644 3272 webserver.cc:460] Webserver started at http://127.3.21.193:44795/ using document root <none> and password file <none>
I20260929 01:00:14.450556 3272 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:14.450724 3272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:14.451081 3272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:14.455232 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "0ff3e44f15e144519e3eb30250933a72"
format_stamp: "Formatted at 2026-09-29 01:00:14 on dist-test-slave-k0b7"
server_key: "4af24ae5cf0ec017be04e741ba6233d5"
server_key_iv: "1e560ca73232f328b572b8ca49e49927"
server_key_version: "encryptionkey@0"
I20260929 01:00:14.456207 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff3e44f15e144519e3eb30250933a72"
format_stamp: "Formatted at 2026-09-29 01:00:14 on dist-test-slave-k0b7"
server_key: "4af24ae5cf0ec017be04e741ba6233d5"
server_key_iv: "1e560ca73232f328b572b8ca49e49927"
server_key_version: "encryptionkey@0"
I20260929 01:00:14.462666 3272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:00:14.467542 3287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:14.469544 3272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:00:14.469739 3272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "0ff3e44f15e144519e3eb30250933a72"
format_stamp: "Formatted at 2026-09-29 01:00:14 on dist-test-slave-k0b7"
server_key: "4af24ae5cf0ec017be04e741ba6233d5"
server_key_iv: "1e560ca73232f328b572b8ca49e49927"
server_key_version: "encryptionkey@0"
I20260929 01:00:14.470005 3272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:14.499734 3272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:14.505051 3272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:14.505409 3272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:14.506649 3272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:14.508118 3272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:14.508205 3272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:14.508343 3272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:14.508407 3272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:14.549791 3272 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:33331
I20260929 01:00:14.549846 3399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:33331 every 8 connection(s)
I20260929 01:00:14.551388 3272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:14.553253 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3272
I20260929 01:00:14.553608 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:14.554402 3159 external_mini_cluster.cc:1428] Setting key 60d860cfe524ea3d942ecd6b904819ff
Sep 29 01:00:14 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:14.579653 3402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:14.554779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:54959 (local address 127.3.21.254:45099)
0929 01:00:14.555696 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:14.555702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:14.558587 (+ 2885us) server_negotiation.cc:407] Connection header received
0929 01:00:14.559244 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:14.559250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:14.559381 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:14.559541 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:14.561413 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:14.562215 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:14.564363 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:14.564815 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:14.564982 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:14.570358 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:14.570366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:14.570371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:14.570425 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:14.574141 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:14.576161 (+ 2020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:14.576169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:14.576173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:14.576457 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:14.577185 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:14.577190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:14.577192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:14.577539 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:14.577669 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:14.579060 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0929 01:00:14.579276 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":700,"threads_started":1}
I20260929 01:00:14.581251 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:14.555668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45099 (local address 127.3.21.193:54959)
0929 01:00:14.556663 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:14.556728 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:14.558882 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:14.559824 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:14.559833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:14.560453 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:14.561188 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:14.561211 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:14.562445 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:14.562451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:14.564159 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:14.564175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:14.565123 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:14.565132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:14.565421 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:14.566410 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:14.566439 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:14.569917 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:14.574596 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:14.574606 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:14.574623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:14.575896 (+ 1273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:14.576635 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:14.576644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:14.576651 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:14.576983 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:14.577731 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:14.577745 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:14.578350 (+ 605us) client_negotiation.cc:769] Sending connection context
0929 01:00:14.578829 (+ 479us) client_negotiation.cc:240] Negotiation successful
0929 01:00:14.579538 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":715,"threads_started":1}
I20260929 01:00:14.583477 3400 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45099
I20260929 01:00:14.583887 3400 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:14.584864 3400 heartbeater.cc:511] Master 127.3.21.254:45099 requested a full tablet report, sending...
I20260929 01:00:14.587847 3211 ts_manager.cc:194] Registered new tserver with Master: 0ff3e44f15e144519e3eb30250933a72 (127.3.21.193:33331)
I20260929 01:00:14.590649 3211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:54959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:14.611770 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45099
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:15.001968 3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:15.002312 3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:15.002422 3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:15.013337 3406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:15.013478 3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:15.013597 3406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:15.024421 3406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:15.025774 3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:15.027206 3406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:15.044163 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:15.044477 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:15.044533 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:15.045228 3406 server_base.cc:1034] running on GCE node
I20260929 01:00:15.045994 3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:15.047405 3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:15.048652 3406 hybrid_clock.cc:648] HybridClock initialized: now 1790643615048576 us; error 83 us; skew 500 ppm
I20260929 01:00:15.049093 3406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:15.053836 3406 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:15.055795 3406 webserver.cc:460] Webserver started at http://127.3.21.194:45097/ using document root <none> and password file <none>
I20260929 01:00:15.056602 3406 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:15.056715 3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:15.057101 3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:15.060482 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "fd63c1daa3594715856344094b9651c8"
format_stamp: "Formatted at 2026-09-29 01:00:15 on dist-test-slave-k0b7"
server_key: "e06e4b1aa61784bac9efed3736bd4dc8"
server_key_iv: "4b7e1d5cd8246c9ea7756ef0f6c61f62"
server_key_version: "encryptionkey@0"
I20260929 01:00:15.061378 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "fd63c1daa3594715856344094b9651c8"
format_stamp: "Formatted at 2026-09-29 01:00:15 on dist-test-slave-k0b7"
server_key: "e06e4b1aa61784bac9efed3736bd4dc8"
server_key_iv: "4b7e1d5cd8246c9ea7756ef0f6c61f62"
server_key_version: "encryptionkey@0"
I20260929 01:00:15.067718 3406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:00:15.072269 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:15.074262 3406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:15.074455 3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "fd63c1daa3594715856344094b9651c8"
format_stamp: "Formatted at 2026-09-29 01:00:15 on dist-test-slave-k0b7"
server_key: "e06e4b1aa61784bac9efed3736bd4dc8"
server_key_iv: "4b7e1d5cd8246c9ea7756ef0f6c61f62"
server_key_version: "encryptionkey@0"
I20260929 01:00:15.074795 3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:15.104100 3406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:15.109599 3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:15.109911 3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:15.111076 3406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:15.112639 3406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:15.112720 3406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:15.112811 3406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:15.112846 3406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:15.163192 3406 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:41301
I20260929 01:00:15.163408 3533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:41301 every 8 connection(s)
I20260929 01:00:15.165033 3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:15.165364 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3406
I20260929 01:00:15.165570 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:15.166111 3159 external_mini_cluster.cc:1428] Setting key ca4461308c3dae90e3c5c71d1c9767e2
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:15.200501 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.170208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:42297 (local address 127.3.21.254:45099)
0929 01:00:15.170904 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:15.170912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:15.172193 (+ 1281us) server_negotiation.cc:407] Connection header received
0929 01:00:15.175455 (+ 3262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:15.175461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:15.175554 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:15.175685 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:15.178405 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.179351 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.182777 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.183312 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.183433 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.192011 (+ 8578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:15.192024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:15.192033 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:15.192097 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:15.196249 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.197671 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.197676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.197678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.197839 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.198419 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.198424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.198426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.198835 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:15.198978 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:15.199967 (+ 989us) server_negotiation.cc:299] Negotiation successful
0929 01:00:15.200175 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":442,"threads_started":1}
I20260929 01:00:15.202785 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.170999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45099 (local address 127.3.21.194:42297)
0929 01:00:15.171986 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:15.172040 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:15.174941 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:15.176032 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:15.176049 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:15.177010 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:15.177996 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:15.178032 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.179564 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.179574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:15.182548 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:15.182578 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.183612 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.183623 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:15.184020 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.185014 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:15.185050 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:15.191613 (+ 6563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:15.196597 (+ 4984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:15.196612 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:15.196637 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:15.197469 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:15.197977 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:15.197985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:15.197992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:15.198277 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:15.198988 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:15.198999 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:15.199500 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 01:00:15.199844 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 01:00:15.200490 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":428,"threads_started":1}
I20260929 01:00:15.204869 3534 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45099
I20260929 01:00:15.205279 3534 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:15.206178 3534 heartbeater.cc:511] Master 127.3.21.254:45099 requested a full tablet report, sending...
I20260929 01:00:15.208226 3211 ts_manager.cc:194] Registered new tserver with Master: fd63c1daa3594715856344094b9651c8 (127.3.21.194:41301)
I20260929 01:00:15.209184 3211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:42297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:15.219775 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45099
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:00:15.594851 3400 heartbeater.cc:503] Master 127.3.21.254:45099 was elected leader, sending a full tablet report...
W20260929 01:00:15.622650 3540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:15.622969 3540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:15.623066 3540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:15.630846 3540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:15.630983 3540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:15.631101 3540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:15.644651 3540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:15.646207 3540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:15.648105 3540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:15.665027 3545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:15.665009 3548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:15.665277 3546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:15.666953 3540 server_base.cc:1034] running on GCE node
I20260929 01:00:15.667886 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:15.669160 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:15.670527 3540 hybrid_clock.cc:648] HybridClock initialized: now 1790643615670372 us; error 123 us; skew 500 ppm
I20260929 01:00:15.670948 3540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:15.675047 3540 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:15.677040 3540 webserver.cc:460] Webserver started at http://127.3.21.195:36793/ using document root <none> and password file <none>
I20260929 01:00:15.677938 3540 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:15.678136 3540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:15.678622 3540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:15.681989 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "905a71df11684322bcbd6c2e32654be4"
format_stamp: "Formatted at 2026-09-29 01:00:15 on dist-test-slave-k0b7"
server_key: "e6c87a3a6ab6388b900e37515aa6f2af"
server_key_iv: "4c1161858aa9a7773984f09202384a87"
server_key_version: "encryptionkey@0"
I20260929 01:00:15.682982 3540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "905a71df11684322bcbd6c2e32654be4"
format_stamp: "Formatted at 2026-09-29 01:00:15 on dist-test-slave-k0b7"
server_key: "e6c87a3a6ab6388b900e37515aa6f2af"
server_key_iv: "4c1161858aa9a7773984f09202384a87"
server_key_version: "encryptionkey@0"
I20260929 01:00:15.689316 3540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 01:00:15.694315 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:15.696173 3540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:00:15.696565 3540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "905a71df11684322bcbd6c2e32654be4"
format_stamp: "Formatted at 2026-09-29 01:00:15 on dist-test-slave-k0b7"
server_key: "e6c87a3a6ab6388b900e37515aa6f2af"
server_key_iv: "4c1161858aa9a7773984f09202384a87"
server_key_version: "encryptionkey@0"
I20260929 01:00:15.696811 3540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:15.720841 3540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:15.726469 3540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:15.726985 3540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:15.728186 3540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:15.730108 3540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:15.730234 3540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:15.730409 3540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:15.730516 3540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:15.775812 3540 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:36873
I20260929 01:00:15.776352 3667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:36873 every 8 connection(s)
I20260929 01:00:15.777534 3540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:15.782549 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3540
I20260929 01:00:15.782882 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:15.783486 3159 external_mini_cluster.cc:1428] Setting key cce25010409c12a1ba241d7b708cd885
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:15.806372 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.781946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:56129 (local address 127.3.21.254:45099)
0929 01:00:15.782919 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:15.782926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:15.783520 (+ 594us) server_negotiation.cc:407] Connection header received
0929 01:00:15.786189 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:15.786194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:15.786356 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:15.786548 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:15.788971 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.789963 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.792768 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.793309 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.793429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.798642 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:15.798648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:15.798651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:15.798690 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:15.802304 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.803531 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.803536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.803538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.803703 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.804312 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.804318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.804320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.804631 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:15.804782 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:15.805730 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 01:00:15.805953 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":568,"threads_started":1}
I20260929 01:00:15.807453 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.782129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45099 (local address 127.3.21.195:56129)
0929 01:00:15.783302 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:15.783363 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:15.785827 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:15.786942 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:15.786960 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:15.787904 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:15.788752 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:15.788777 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.790148 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.790154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:15.792539 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:15.792565 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.793569 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.793576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:15.793929 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.794838 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:15.794870 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:15.798320 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:15.802571 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:15.802595 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:15.802620 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:15.803195 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:15.803867 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:15.803873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:15.803876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:15.804114 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:15.804788 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:15.804796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:15.805223 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 01:00:15.805526 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 01:00:15.805995 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":575,"threads_started":1}
I20260929 01:00:15.809477 3668 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45099
I20260929 01:00:15.809939 3668 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:15.810835 3668 heartbeater.cc:511] Master 127.3.21.254:45099 requested a full tablet report, sending...
I20260929 01:00:15.812635 3211 ts_manager.cc:194] Registered new tserver with Master: 905a71df11684322bcbd6c2e32654be4 (127.3.21.195:36873)
I20260929 01:00:15.813767 3211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:56129
I20260929 01:00:15.814244 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:15.869755 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.848244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.3.21.193:33331)
0929 01:00:15.849292 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:15.849304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:15.849338 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:00:15.849422 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:15.849429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:15.849664 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:15.849828 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:15.852136 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.853451 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.855473 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.856632 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.856780 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.861901 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:15.861918 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:15.861935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:15.861990 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:15.865747 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.866671 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.866682 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.866685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.866858 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.867341 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.867345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.867348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.867775 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:15.868056 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:15.868909 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 01:00:15.869173 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":537,"threads_started":1}
W20260929 01:00:15.871219 3315 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45598
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:00:15 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:15.924664 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.907463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.3.21.193:33331)
0929 01:00:15.907714 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:15.907730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:15.907933 (+ 203us) server_negotiation.cc:407] Connection header received
0929 01:00:15.908006 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:15.908010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:15.908250 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:15.908357 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:15.910120 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.911142 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.912939 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.913467 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.913574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.918327 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:15.918334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:15.918338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:15.918382 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:15.921658 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.922609 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.922615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.922619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.922778 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.923182 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.923186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.923188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.923446 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:15.923620 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:15.924035 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 01:00:15.924315 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 01:00:15.926447 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3272
I20260929 01:00:15.942145 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:15.927508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.3.21.193:33331)
0929 01:00:15.927703 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:15.927707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:15.928064 (+ 357us) server_negotiation.cc:407] Connection header received
0929 01:00:15.928350 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:15.928354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:15.928415 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:15.928535 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:15.929926 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.930737 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.932078 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:15.932545 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:15.932648 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:15.935998 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:15.936005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:15.936009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:15.936052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:15.939416 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.940255 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.940262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.940266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.940420 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:15.940752 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:15.940755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:15.940758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:15.941043 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:15.941330 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:15.941599 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 01:00:15.941856 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:00:15.943032 3395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:16.042822 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3272
I20260929 01:00:16.061656 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3406
Sep 29 01:00:16 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:16.085071 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:16.063243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.3.21.194:41301)
0929 01:00:16.064362 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:16.064370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:16.064404 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:00:16.064468 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:16.064475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:16.064723 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:16.064917 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:16.066408 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:16.067637 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:16.069139 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:16.069812 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:16.070025 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:16.076671 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:16.076692 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:16.076711 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:16.076759 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:16.081585 (+ 4826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:16.082536 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:16.082554 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:16.082557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:16.082721 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:16.083186 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:16.083194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:16.083203 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:16.083604 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:16.083973 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:16.084342 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 01:00:16.084666 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":632,"threads_started":1}
I20260929 01:00:16.087661 3529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:16.177896 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3406
I20260929 01:00:16.201607 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3540
Sep 29 01:00:16 dist-test-slave-k0b7 krb5kdc[3166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:16.223165 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:16.203008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.3.21.195:36873)
0929 01:00:16.203667 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:16.203675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:16.203708 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:00:16.203781 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:16.203788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:16.204028 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:16.204162 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:16.205768 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:16.207170 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:16.208853 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:16.209562 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:16.209676 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:16.214794 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:16.214820 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:16.214858 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:16.214940 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:16.219088 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:16.220253 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:16.220264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:16.220272 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:16.220441 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:16.221111 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:16.221117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:16.221121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:16.221562 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:16.221906 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:16.222323 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 01:00:16.222726 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":374,"threads_started":1}
I20260929 01:00:16.224852 3663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:16.319126 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3540
I20260929 01:00:16.337528 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3179
I20260929 01:00:16.339025 3241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:16.451339 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3179
2026-09-29T01:00:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4558 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:16 dist-test-slave-k0b7 krb5kdc[3690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:16 dist-test-slave-k0b7 krb5kdc[3690](info): set up 2 sockets
Sep 29 01:00:16 dist-test-slave-k0b7 krb5kdc[3690](info): commencing operation
krb5kdc: starting...
W20260929 01:00:17.619011 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:17 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:17.843560 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:38049
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:18.220317 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:18.220692 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:18.220788 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:18.228471 3703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:18.228612 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:18.228660 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:18.228711 3703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:18.228752 3703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:18.239631 3703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:38049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:18.241379 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:18.243002 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:18.258437 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:18.258553 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:18.258515 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:18.259559 3703 server_base.cc:1034] running on GCE node
I20260929 01:00:18.260736 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:18.262593 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:18.263893 3703 hybrid_clock.cc:648] HybridClock initialized: now 1790643618263840 us; error 56 us; skew 500 ppm
I20260929 01:00:18.264485 3703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:18 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:18.269505 3703 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:18.271842 3703 webserver.cc:460] Webserver started at http://127.3.21.254:39031/ using document root <none> and password file <none>
I20260929 01:00:18.272676 3703 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:18.272800 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:18.273078 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:18.276472 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "f5bb3365aaef421487b6a3e6e74ae79c"
format_stamp: "Formatted at 2026-09-29 01:00:18 on dist-test-slave-k0b7"
server_key: "8cf13059072dafabbc137d5521a57e9f"
server_key_iv: "208c3111eddb7a59ac73f35ae9c89357"
server_key_version: "encryptionkey@0"
I20260929 01:00:18.277271 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "f5bb3365aaef421487b6a3e6e74ae79c"
format_stamp: "Formatted at 2026-09-29 01:00:18 on dist-test-slave-k0b7"
server_key: "8cf13059072dafabbc137d5521a57e9f"
server_key_iv: "208c3111eddb7a59ac73f35ae9c89357"
server_key_version: "encryptionkey@0"
I20260929 01:00:18.283915 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 01:00:18.288318 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:18.290335 3703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:18.290624 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "f5bb3365aaef421487b6a3e6e74ae79c"
format_stamp: "Formatted at 2026-09-29 01:00:18 on dist-test-slave-k0b7"
server_key: "8cf13059072dafabbc137d5521a57e9f"
server_key_iv: "208c3111eddb7a59ac73f35ae9c89357"
server_key_version: "encryptionkey@0"
I20260929 01:00:18.290841 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:18.308615 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:18.314920 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:18.315215 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:18.338320 3703 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:38049
I20260929 01:00:18.338356 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:38049 every 8 connection(s)
I20260929 01:00:18.340104 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:18.345083 3770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:18.346588 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3703
I20260929 01:00:18.346850 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:18.347491 3159 external_mini_cluster.cc:1428] Setting key a6db1a732d0785819639577f0b8f54b5
I20260929 01:00:18.358353 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c: Bootstrap starting.
I20260929 01:00:18.362768 3770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c: Neither blocks nor log segments found. Creating new log.
Sep 29 01:00:18 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:18.364066 3770 log.cc:824] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:18.369004 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c: No bootstrap required, opened a new log
I20260929 01:00:18.377434 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:18.349677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.3.21.254:38049)
0929 01:00:18.350677 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:18.350691 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:18.350750 (+ 59us) server_negotiation.cc:407] Connection header received
0929 01:00:18.352608 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:18.352634 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:18.353102 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:18.353481 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:18.355424 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:18.356987 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:18.358803 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:18.359653 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:18.359824 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:18.365353 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:18.365364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:18.365383 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:18.365430 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:18.371497 (+ 6067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:18.372496 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:18.372508 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:18.372523 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:18.372799 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:18.373355 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:18.373361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:18.373365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:18.374238 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:18.374580 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:18.375279 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 01:00:18.375834 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":443,"threads_started":1}
I20260929 01:00:18.380553 3770 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38049 } }
I20260929 01:00:18.381053 3770 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:18.381182 3770 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bb3365aaef421487b6a3e6e74ae79c, State: Initialized, Role: FOLLOWER
I20260929 01:00:18.381897 3770 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38049 } }
I20260929 01:00:18.382157 3770 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:18.382292 3770 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:18.382467 3770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:18.384848 3770 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38049 } }
I20260929 01:00:18.385468 3770 leader_election.cc:302] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5bb3365aaef421487b6a3e6e74ae79c; no voters:
I20260929 01:00:18.386197 3770 leader_election.cc:288] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:18.386420 3775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:18.387749 3775 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [term 1 LEADER]: Becoming Leader. State: Replica: f5bb3365aaef421487b6a3e6e74ae79c, State: Running, Role: LEADER
I20260929 01:00:18.388415 3775 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38049 } }
I20260929 01:00:18.388666 3770 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:18.391657 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38049 } } }
I20260929 01:00:18.392040 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:18.391443 3777 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5bb3365aaef421487b6a3e6e74ae79c. Latest consensus state: current_term: 1 leader_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5bb3365aaef421487b6a3e6e74ae79c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38049 } } }
I20260929 01:00:18.392750 3777 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:18.394722 3784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:18.400589 3784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:18.410312 3784 catalog_manager.cc:1347] Generated new cluster ID: 4c69023c1d0d428a98955d062752d2bc
I20260929 01:00:18.410456 3784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:18.423710 3784 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:18.426090 3784 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:18.441649 3784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5bb3365aaef421487b6a3e6e74ae79c: Generated new TSK 0
I20260929 01:00:18.442893 3784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:18.499334 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38049
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:18.886760 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:18.887074 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:18.887282 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:18.895785 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:18.895911 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:18.896032 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:18.908589 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:18.910036 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:18.911656 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:18.929472 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:18.929618 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:18.929625 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:18.930568 3796 server_base.cc:1034] running on GCE node
I20260929 01:00:18.931380 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:18.932585 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:18.933876 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790643618933834 us; error 74 us; skew 500 ppm
I20260929 01:00:18.934316 3796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:18 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:18.947037 3796 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:18.948860 3796 webserver.cc:460] Webserver started at http://127.3.21.193:37275/ using document root <none> and password file <none>
I20260929 01:00:18.949606 3796 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:18.949719 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:18.949991 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:18.953271 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1"
format_stamp: "Formatted at 2026-09-29 01:00:18 on dist-test-slave-k0b7"
server_key: "db5fd5960a0d7928ae459276918578d8"
server_key_iv: "3d7751489e27d661a86dc166df9c8ef2"
server_key_version: "encryptionkey@0"
I20260929 01:00:18.954123 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1"
format_stamp: "Formatted at 2026-09-29 01:00:18 on dist-test-slave-k0b7"
server_key: "db5fd5960a0d7928ae459276918578d8"
server_key_iv: "3d7751489e27d661a86dc166df9c8ef2"
server_key_version: "encryptionkey@0"
I20260929 01:00:18.960216 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:00:18.964979 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:18.967947 3796 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:00:18.968364 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1"
format_stamp: "Formatted at 2026-09-29 01:00:18 on dist-test-slave-k0b7"
server_key: "db5fd5960a0d7928ae459276918578d8"
server_key_iv: "3d7751489e27d661a86dc166df9c8ef2"
server_key_version: "encryptionkey@0"
I20260929 01:00:18.968624 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:19.010648 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:19.015471 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:19.015815 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:19.016950 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:19.019934 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:19.020148 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:19.020417 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:19.020500 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:19.066052 3796 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:39541
I20260929 01:00:19.066100 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:39541 every 8 connection(s)
I20260929 01:00:19.068188 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:19.072096 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3796
I20260929 01:00:19.072364 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:19.073423 3159 external_mini_cluster.cc:1428] Setting key f175ffbc20275302846fb85cbbaf52f2
Sep 29 01:00:19 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:19.105433 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:19.073003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:52305 (local address 127.3.21.254:38049)
0929 01:00:19.073997 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:19.074004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:19.075832 (+ 1828us) server_negotiation.cc:407] Connection header received
0929 01:00:19.078017 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:19.078026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:19.078146 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:19.078307 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:19.081143 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.082270 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.085483 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.086332 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.086676 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:19.093604 (+ 6928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:19.093616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:19.093623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:19.093687 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:19.098922 (+ 5235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:19.100850 (+ 1928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:19.100856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:19.100859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:19.101085 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:19.102056 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:19.102063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:19.102067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:19.102613 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:19.102816 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:19.104113 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0929 01:00:19.104901 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":519,"threads_started":1}
I20260929 01:00:19.106637 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:19.074376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38049 (local address 127.3.21.193:52305)
0929 01:00:19.075633 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:19.075699 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:19.077652 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:19.078858 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:19.078876 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:19.079818 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:19.080830 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:19.080866 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.082612 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.082622 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:19.085190 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:19.085218 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.086776 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.086790 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:19.087209 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:19.088255 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:19.088297 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:19.093191 (+ 4894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:19.099368 (+ 6177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:19.099382 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:19.099420 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:19.100402 (+ 982us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:19.101402 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:19.101413 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:19.101420 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:19.101764 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:19.102951 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:19.102959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:19.103517 (+ 558us) client_negotiation.cc:769] Sending connection context
0929 01:00:19.103832 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 01:00:19.104678 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":480,"threads_started":1}
I20260929 01:00:19.108625 3924 heartbeater.cc:347] Connected to a master server at 127.3.21.254:38049
I20260929 01:00:19.109012 3924 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:19.109993 3924 heartbeater.cc:511] Master 127.3.21.254:38049 requested a full tablet report, sending...
I20260929 01:00:19.113436 3735 ts_manager.cc:194] Registered new tserver with Master: e87dcb91a9c242e08ee5e1fbe0d531f1 (127.3.21.193:39541)
I20260929 01:00:19.117231 3735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:52305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:19.132841 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38049
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:19.524823 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:19.525218 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:19.525384 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:19.533607 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:19.533794 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:19.533968 3930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:19.544860 3930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:19.546283 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:19.547814 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:19.562729 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:19.562649 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:19.562707 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:19.564303 3930 server_base.cc:1034] running on GCE node
I20260929 01:00:19.565150 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:19.566387 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:19.567683 3930 hybrid_clock.cc:648] HybridClock initialized: now 1790643619567625 us; error 54 us; skew 500 ppm
I20260929 01:00:19.568087 3930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:19 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:19.572050 3930 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:19.574386 3930 webserver.cc:460] Webserver started at http://127.3.21.194:41467/ using document root <none> and password file <none>
I20260929 01:00:19.575524 3930 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:19.575742 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:19.576102 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:19.579448 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "69e9c175a87e4e9390abaf8a2f387e0d"
format_stamp: "Formatted at 2026-09-29 01:00:19 on dist-test-slave-k0b7"
server_key: "3f16cb25d0f79253ea15f5b455aa499b"
server_key_iv: "bfb0548feeba6fbafc6c0b006c80c91d"
server_key_version: "encryptionkey@0"
I20260929 01:00:19.580328 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "69e9c175a87e4e9390abaf8a2f387e0d"
format_stamp: "Formatted at 2026-09-29 01:00:19 on dist-test-slave-k0b7"
server_key: "3f16cb25d0f79253ea15f5b455aa499b"
server_key_iv: "bfb0548feeba6fbafc6c0b006c80c91d"
server_key_version: "encryptionkey@0"
I20260929 01:00:19.586472 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:00:19.591007 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:19.592849 3930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:00:19.593204 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "69e9c175a87e4e9390abaf8a2f387e0d"
format_stamp: "Formatted at 2026-09-29 01:00:19 on dist-test-slave-k0b7"
server_key: "3f16cb25d0f79253ea15f5b455aa499b"
server_key_iv: "bfb0548feeba6fbafc6c0b006c80c91d"
server_key_version: "encryptionkey@0"
I20260929 01:00:19.593451 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:19.616667 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:19.621528 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:19.621843 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:19.622936 3930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:19.624502 3930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:19.624641 3930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:19.624826 3930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:19.624899 3930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:19.664916 3930 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:41621
I20260929 01:00:19.664976 4057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:41621 every 8 connection(s)
I20260929 01:00:19.666373 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:19.675884 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 3930
I20260929 01:00:19.676086 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:19.676608 3159 external_mini_cluster.cc:1428] Setting key 153ce10ffaddb879c03fdf9e7f8063b1
Sep 29 01:00:19 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643619, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:19.692792 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:19.669578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:39771 (local address 127.3.21.254:38049)
0929 01:00:19.670064 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:19.670069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:19.671298 (+ 1229us) server_negotiation.cc:407] Connection header received
0929 01:00:19.673354 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:19.673360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:19.673473 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:19.673603 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:19.675296 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.676399 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.678916 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.679730 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.679886 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:19.684364 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:19.684374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:19.684382 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:19.684436 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:19.688595 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:19.689762 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:19.689771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:19.689777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:19.689964 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:19.690793 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:19.690809 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:19.690816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:19.691119 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:19.691281 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:19.692178 (+ 897us) server_negotiation.cc:299] Negotiation successful
0929 01:00:19.692421 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260929 01:00:19.693988 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:19.670229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38049 (local address 127.3.21.194:39771)
0929 01:00:19.671129 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:19.671170 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:19.673004 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:19.673892 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:19.673902 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:19.674440 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:19.675111 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:19.675130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.676620 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.676625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:19.678717 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:19.678737 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:19.679986 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:19.679991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:19.680352 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:19.681153 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:19.681184 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:19.683941 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:19.688884 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:19.688897 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:19.688925 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:19.689586 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:19.690096 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:19.690101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:19.690105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:19.690645 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:19.691256 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:19.691266 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:19.691698 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 01:00:19.691978 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 01:00:19.692582 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":424,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:00:19.696102 4058 heartbeater.cc:347] Connected to a master server at 127.3.21.254:38049
I20260929 01:00:19.696508 4058 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:19.697336 4058 heartbeater.cc:511] Master 127.3.21.254:38049 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:00:19.699405 3735 ts_manager.cc:194] Registered new tserver with Master: 69e9c175a87e4e9390abaf8a2f387e0d (127.3.21.194:41621)
I20260929 01:00:19.700310 3735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:39771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:19.728185 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38049
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:20.105772 4064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:20.106102 4064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:20.106263 4064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:20.113983 4064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:20.114130 4064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:20.114316 4064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:20.122849 3924 heartbeater.cc:503] Master 127.3.21.254:38049 was elected leader, sending a full tablet report...
I20260929 01:00:20.130146 4064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:20.133417 4064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:20.136639 4064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:20.157094 4069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:20.157482 4070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:20.157047 4072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:20.157881 4064 server_base.cc:1034] running on GCE node
I20260929 01:00:20.158712 4064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:20.159797 4064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:20.161087 4064 hybrid_clock.cc:648] HybridClock initialized: now 1790643620161003 us; error 74 us; skew 500 ppm
I20260929 01:00:20.161468 4064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:20.165481 4064 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:20.167371 4064 webserver.cc:460] Webserver started at http://127.3.21.195:42179/ using document root <none> and password file <none>
I20260929 01:00:20.168159 4064 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:20.168314 4064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:20.168651 4064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:20.172634 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "28e8f51bb33d4b3680050125d13fbfea"
format_stamp: "Formatted at 2026-09-29 01:00:20 on dist-test-slave-k0b7"
server_key: "3ec2c44bb40c896e49dee2bc67d166ab"
server_key_iv: "6b79aefbc2f7452e718772de8ef2f544"
server_key_version: "encryptionkey@0"
I20260929 01:00:20.173801 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "28e8f51bb33d4b3680050125d13fbfea"
format_stamp: "Formatted at 2026-09-29 01:00:20 on dist-test-slave-k0b7"
server_key: "3ec2c44bb40c896e49dee2bc67d166ab"
server_key_iv: "6b79aefbc2f7452e718772de8ef2f544"
server_key_version: "encryptionkey@0"
I20260929 01:00:20.180536 4064 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 01:00:20.186740 4079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:20.189357 4064 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260929 01:00:20.189646 4064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "28e8f51bb33d4b3680050125d13fbfea"
format_stamp: "Formatted at 2026-09-29 01:00:20 on dist-test-slave-k0b7"
server_key: "3ec2c44bb40c896e49dee2bc67d166ab"
server_key_iv: "6b79aefbc2f7452e718772de8ef2f544"
server_key_version: "encryptionkey@0"
I20260929 01:00:20.190028 4064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:20.224573 4064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:20.230393 4064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:20.230758 4064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:20.232103 4064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:20.233584 4064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:20.233690 4064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:20.233824 4064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:20.233894 4064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:20.276674 4064 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:33763
I20260929 01:00:20.276763 4191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:33763 every 8 connection(s)
I20260929 01:00:20.278033 4064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:20.281636 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4064
I20260929 01:00:20.281864 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:20.282428 3159 external_mini_cluster.cc:1428] Setting key 14e8ee619e26a34463f4c8964dfb4c81
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:20.306075 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.281382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:52745 (local address 127.3.21.254:38049)
0929 01:00:20.282039 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.282046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.283301 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 01:00:20.285996 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.286010 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.286088 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.286207 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.288417 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.289893 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.292184 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.292636 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.292796 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.298426 (+ 5630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.298432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.298435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.298475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.301689 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.302650 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.302656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.302660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.302801 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.303307 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.303311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.303314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.303622 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.303791 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.305186 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.305494 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":307,"threads_started":1}
I20260929 01:00:20.307420 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.282138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38049 (local address 127.3.21.195:52745)
0929 01:00:20.283121 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:20.283175 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:20.285650 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:20.286727 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:20.286736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:20.287365 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:20.288064 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.288087 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.290375 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.290381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.291969 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.291987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.292923 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.292931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.293241 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.294105 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:20.294132 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:20.298119 (+ 3987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:20.301886 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:20.301898 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:20.301921 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:20.302452 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:20.302924 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:20.302929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:20.302933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:20.303169 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:20.303760 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:20.303769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:20.304325 (+ 556us) client_negotiation.cc:769] Sending connection context
0929 01:00:20.305191 (+ 866us) client_negotiation.cc:240] Negotiation successful
0929 01:00:20.305905 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":436,"threads_started":1}
I20260929 01:00:20.309442 4192 heartbeater.cc:347] Connected to a master server at 127.3.21.254:38049
I20260929 01:00:20.309821 4192 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:20.310906 4192 heartbeater.cc:511] Master 127.3.21.254:38049 requested a full tablet report, sending...
I20260929 01:00:20.312767 3735 ts_manager.cc:194] Registered new tserver with Master: 28e8f51bb33d4b3680050125d13fbfea (127.3.21.195:33763)
I20260929 01:00:20.313706 3735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:52745
I20260929 01:00:20.325063 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:20.348836 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.332881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.3.21.254:38049)
0929 01:00:20.333082 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.333088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.333689 (+ 601us) server_negotiation.cc:407] Connection header received
0929 01:00:20.333929 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.333933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.334005 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.334383 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.336022 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.336802 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.338269 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.338817 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.338957 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.341852 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.341858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.341862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.341907 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.345299 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.346117 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.346122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.346131 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.346262 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.346790 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.346798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.346802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.347161 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.347432 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.347789 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.348055 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:00:20.357535 3735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:20.364943 3735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:20.406040 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.380784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:41621 (local address 127.0.0.1:35402)
0929 01:00:20.394825 (+ 14041us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:20.394848 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:20.395041 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:20.396062 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:20.396071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:20.396123 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:20.396971 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.396988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.399812 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.399817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.402805 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.402821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.404663 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.404673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.404922 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.404989 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 01:00:20.405169 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 01:00:20.405429 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13903,"mutex_wait_us":69,"spinlock_wait_cycles":7424,"thread_start_us":13563,"threads_started":1}
I20260929 01:00:20.406881 4209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.380766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:33763 (local address 127.0.0.1:38914)
0929 01:00:20.392291 (+ 11525us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:20.392341 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:20.392593 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:20.396292 (+ 3699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:20.396299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:20.396344 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:20.397324 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.397338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.399530 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.399540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.403717 (+ 4177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.403732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.405897 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.405903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.406080 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.406130 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 01:00:20.406243 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 01:00:20.406408 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11246,"mutex_wait_us":3,"thread_start_us":14499,"threads_started":1}
I20260929 01:00:20.407212 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.381558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.3.21.194:41621)
0929 01:00:20.394850 (+ 13292us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.394863 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.395214 (+ 351us) server_negotiation.cc:407] Connection header received
0929 01:00:20.395364 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.395373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.395774 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.397402 (+ 1628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:20.397678 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.399622 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.403017 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.404354 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.404540 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.404799 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.405260 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.405504 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13112,"thread_start_us":12984,"threads_started":1}
I20260929 01:00:20.407902 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.3.21.195:33763)
0929 01:00:20.395259 (+ 13091us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.395269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.395300 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:00:20.395377 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.395383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.395699 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.396091 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:20.397737 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.399342 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.403910 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.405330 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.405472 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.405716 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.407219 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.407438 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12921,"thread_start_us":12742,"threads_started":1}
I20260929 01:00:20.408496 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.382055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:39541 (local address 127.0.0.1:48826)
0929 01:00:20.396595 (+ 14540us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:20.396620 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:20.396856 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:20.398386 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:20.398392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:20.398428 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:20.398910 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.398923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.400682 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.400689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.403548 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.403565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.405949 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.405960 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.406563 (+ 603us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.406642 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 01:00:20.406729 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 01:00:20.406887 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13860,"mutex_wait_us":1,"thread_start_us":9763,"threads_started":1}
I20260929 01:00:20.409490 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.381620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.3.21.193:39541)
0929 01:00:20.391590 (+ 9970us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.391606 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.397085 (+ 5479us) server_negotiation.cc:407] Connection header received
0929 01:00:20.397337 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.397352 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.397716 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.397951 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:20.399105 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.400511 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.403822 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.405360 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.405503 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.405762 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.408859 (+ 3097us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.409087 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9717,"thread_start_us":9487,"threads_started":1}
I20260929 01:00:20.411621 3993 tablet_service.cc:1467] Processing CreateTablet for tablet 0848f1598f6b421b84e11c7f67b4b3ce (DEFAULT_TABLE table=test-table [id=be47cc70889c48178252325fdc2adaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:20.412235 4127 tablet_service.cc:1467] Processing CreateTablet for tablet 0848f1598f6b421b84e11c7f67b4b3ce (DEFAULT_TABLE table=test-table [id=be47cc70889c48178252325fdc2adaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:20.413962 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0848f1598f6b421b84e11c7f67b4b3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:20.413630 3859 tablet_service.cc:1467] Processing CreateTablet for tablet 0848f1598f6b421b84e11c7f67b4b3ce (DEFAULT_TABLE table=test-table [id=be47cc70889c48178252325fdc2adaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:20.414602 4127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0848f1598f6b421b84e11c7f67b4b3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:20.415491 3859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0848f1598f6b421b84e11c7f67b4b3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:20.439239 4211 tablet_bootstrap.cc:492] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: Bootstrap starting.
I20260929 01:00:20.439460 4213 tablet_bootstrap.cc:492] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: Bootstrap starting.
I20260929 01:00:20.442060 4212 tablet_bootstrap.cc:492] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: Bootstrap starting.
I20260929 01:00:20.444073 4211 tablet_bootstrap.cc:654] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:20.444257 4213 tablet_bootstrap.cc:654] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:20.445317 4213 log.cc:824] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:20.445441 4211 log.cc:824] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:20.446022 4212 tablet_bootstrap.cc:654] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:20.447789 4212 log.cc:824] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:20.452682 4211 tablet_bootstrap.cc:492] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: No bootstrap required, opened a new log
I20260929 01:00:20.452903 4212 tablet_bootstrap.cc:492] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: No bootstrap required, opened a new log
I20260929 01:00:20.453047 4211 ts_tablet_manager.cc:1397] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260929 01:00:20.453159 4212 ts_tablet_manager.cc:1397] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260929 01:00:20.457156 4213 tablet_bootstrap.cc:492] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: No bootstrap required, opened a new log
I20260929 01:00:20.457655 4213 ts_tablet_manager.cc:1397] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260929 01:00:20.465688 4212 raft_consensus.cc:348] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.465688 4211 raft_consensus.cc:348] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.466540 4212 raft_consensus.cc:374] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:20.466588 4211 raft_consensus.cc:374] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:20.466703 4211 raft_consensus.cc:729] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87dcb91a9c242e08ee5e1fbe0d531f1, State: Initialized, Role: FOLLOWER
I20260929 01:00:20.466703 4212 raft_consensus.cc:729] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e9c175a87e4e9390abaf8a2f387e0d, State: Initialized, Role: FOLLOWER
I20260929 01:00:20.467617 4212 consensus_queue.cc:260] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.467625 4211 consensus_queue.cc:260] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.477268 4211 ts_tablet_manager.cc:1428] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: Time spent starting tablet: real 0.024s user 0.013s sys 0.010s
I20260929 01:00:20.476653 4213 raft_consensus.cc:348] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.478132 4213 raft_consensus.cc:374] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:20.478348 4213 raft_consensus.cc:729] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8f51bb33d4b3680050125d13fbfea, State: Initialized, Role: FOLLOWER
I20260929 01:00:20.478367 4058 heartbeater.cc:503] Master 127.3.21.254:38049 was elected leader, sending a full tablet report...
I20260929 01:00:20.479162 4212 ts_tablet_manager.cc:1428] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: Time spent starting tablet: real 0.026s user 0.022s sys 0.006s
I20260929 01:00:20.479669 4213 consensus_queue.cc:260] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.481006 4192 heartbeater.cc:503] Master 127.3.21.254:38049 was elected leader, sending a full tablet report...
I20260929 01:00:20.481760 4213 ts_tablet_manager.cc:1428] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: Time spent starting tablet: real 0.024s user 0.014s sys 0.006s
W20260929 01:00:20.530099 4193 tablet.cc:2369] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:20.572185 3925 tablet.cc:2369] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:20.572264 4218 raft_consensus.cc:482] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:20.572649 4218 raft_consensus.cc:504] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.574922 4218 leader_election.cc:288] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28e8f51bb33d4b3680050125d13fbfea (127.3.21.195:33763), 69e9c175a87e4e9390abaf8a2f387e0d (127.3.21.194:41621)
I20260929 01:00:20.583303 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.575331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:53801 (local address 127.3.21.195:33763)
0929 01:00:20.575568 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.575577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.576700 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 01:00:20.576802 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.576807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.576972 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.577087 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:20.577921 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.578896 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.581443 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.582261 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.582472 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.582586 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.582896 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.583026 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:00:20.583523 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.575893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:33659 (local address 127.3.21.194:41621)
0929 01:00:20.576116 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.576130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.577229 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 01:00:20.577375 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.577380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.577465 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.577580 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:20.578415 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.579509 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.581687 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.582477 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.582735 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.582809 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.583101 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.583234 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:00:20.583623 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.575661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:41621 (local address 127.3.21.193:33659)
0929 01:00:20.577094 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:20.577114 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:20.577237 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:20.577628 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:20.577633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:20.577658 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:20.578061 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.578072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.579725 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.579736 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.581512 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.581525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.582755 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.582762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.582917 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.582941 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:00:20.583074 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 01:00:20.583217 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"mutex_wait_us":6,"thread_start_us":993,"threads_started":1}
I20260929 01:00:20.583961 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.575157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:33763 (local address 127.3.21.193:53801)
0929 01:00:20.576098 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:20.576131 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:20.576361 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:20.577185 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:20.577193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:20.577309 (+ 116us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:20.577747 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.577762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.579125 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.579131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.581232 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:20.581254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.582523 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.582530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:20.582704 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.582745 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:00:20.582860 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 01:00:20.583012 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":83,"spinlock_wait_cycles":25472,"thread_start_us":498,"threads_started":1}
I20260929 01:00:20.584970 4013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0848f1598f6b421b84e11c7f67b4b3ce" candidate_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" is_pre_election: true
I20260929 01:00:20.585058 4147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0848f1598f6b421b84e11c7f67b4b3ce" candidate_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e8f51bb33d4b3680050125d13fbfea" is_pre_election: true
I20260929 01:00:20.585610 4013 raft_consensus.cc:2454] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87dcb91a9c242e08ee5e1fbe0d531f1 in term 0.
I20260929 01:00:20.585613 4147 raft_consensus.cc:2454] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87dcb91a9c242e08ee5e1fbe0d531f1 in term 0.
I20260929 01:00:20.586416 3813 leader_election.cc:302] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28e8f51bb33d4b3680050125d13fbfea, 69e9c175a87e4e9390abaf8a2f387e0d, e87dcb91a9c242e08ee5e1fbe0d531f1; no voters:
I20260929 01:00:20.587100 4218 raft_consensus.cc:2781] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:20.587277 4218 raft_consensus.cc:482] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:20.587389 4218 raft_consensus.cc:3037] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:20.590638 4218 raft_consensus.cc:504] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.591336 4218 leader_election.cc:288] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [CANDIDATE]: Term 1 election: Requested vote from peers 28e8f51bb33d4b3680050125d13fbfea (127.3.21.195:33763), 69e9c175a87e4e9390abaf8a2f387e0d (127.3.21.194:41621)
I20260929 01:00:20.592067 4013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0848f1598f6b421b84e11c7f67b4b3ce" candidate_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e9c175a87e4e9390abaf8a2f387e0d"
I20260929 01:00:20.592111 4147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0848f1598f6b421b84e11c7f67b4b3ce" candidate_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28e8f51bb33d4b3680050125d13fbfea"
I20260929 01:00:20.592320 4013 raft_consensus.cc:3037] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:20.592355 4147 raft_consensus.cc:3037] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:20.595032 4013 raft_consensus.cc:2454] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87dcb91a9c242e08ee5e1fbe0d531f1 in term 1.
I20260929 01:00:20.595183 4147 raft_consensus.cc:2454] T 0848f1598f6b421b84e11c7f67b4b3ce P 28e8f51bb33d4b3680050125d13fbfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87dcb91a9c242e08ee5e1fbe0d531f1 in term 1.
I20260929 01:00:20.595840 3815 leader_election.cc:302] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28e8f51bb33d4b3680050125d13fbfea, e87dcb91a9c242e08ee5e1fbe0d531f1; no voters:
I20260929 01:00:20.596369 4218 raft_consensus.cc:2781] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:20.596969 4218 raft_consensus.cc:686] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [term 1 LEADER]: Becoming Leader. State: Replica: e87dcb91a9c242e08ee5e1fbe0d531f1, State: Running, Role: LEADER
I20260929 01:00:20.597568 4218 consensus_queue.cc:237] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } }
I20260929 01:00:20.604714 3734 catalog_manager.cc:5495] T 0848f1598f6b421b84e11c7f67b4b3ce P e87dcb91a9c242e08ee5e1fbe0d531f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e87dcb91a9c242e08ee5e1fbe0d531f1 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87dcb91a9c242e08ee5e1fbe0d531f1" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28e8f51bb33d4b3680050125d13fbfea" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e9c175a87e4e9390abaf8a2f387e0d" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41621 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:20.663905 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.645874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.3.21.193:39541)
0929 01:00:20.646162 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.646166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.647371 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 01:00:20.647663 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.647668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.647742 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.647860 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.649546 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.650699 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.652283 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.652701 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.652848 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.657336 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.657355 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.657372 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.657422 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.660738 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.661626 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.661634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.661636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.661791 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.662216 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.662221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.662225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.662691 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.663007 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.663308 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.663564 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 01:00:20.669308 4059 tablet.cc:2369] T 0848f1598f6b421b84e11c7f67b4b3ce P 69e9c175a87e4e9390abaf8a2f387e0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:20.716919 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.701101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.3.21.193:39541)
0929 01:00:20.701381 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.701388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.701636 (+ 248us) server_negotiation.cc:407] Connection header received
0929 01:00:20.701949 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.701957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.702046 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.702365 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.703885 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.704667 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.705988 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.706426 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.706634 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.710908 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.710914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.710917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.710959 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.713879 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.714859 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.714866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.714870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.715031 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.715475 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.715481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.715484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.715768 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.716077 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.716326 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.716574 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:20.771626 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.751945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.3.21.193:39541)
0929 01:00:20.752203 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.752210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.752554 (+ 344us) server_negotiation.cc:407] Connection header received
0929 01:00:20.752775 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.752779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.752865 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.752962 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.754630 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.755483 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.757035 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.757552 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.757668 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.762647 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.762658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.762665 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.762729 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.767296 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.769170 (+ 1874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.769179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.769185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.769385 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.769880 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.769890 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.769893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.770349 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.770685 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.771020 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.771310 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 01:00:20.773581 3839 tablet_service.cc:2908] Rejecting scan request for tablet 0848f1598f6b421b84e11c7f67b4b3ce: Uninitialized: clean time has not yet been initialized
I20260929 01:00:20.777379 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3796
I20260929 01:00:20.794185 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.778399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.3.21.193:39541)
0929 01:00:20.778718 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.778728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.779210 (+ 482us) server_negotiation.cc:407] Connection header received
0929 01:00:20.779290 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.779294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.779402 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.779555 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.781010 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.781802 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.783306 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.783799 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.783914 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.787028 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.787035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.787039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.787082 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.790262 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.791575 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.791582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.791585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.791764 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.792334 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.792338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.792341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.792657 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.792953 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.793328 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.793735 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 01:00:20.795393 3919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:20.896309 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3796
I20260929 01:00:20.919378 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3930
Sep 29 01:00:20 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:20.941490 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:20.920727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.3.21.194:41621)
0929 01:00:20.920898 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:20.920903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:20.920925 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:00:20.921092 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:20.921096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:20.921207 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:20.921320 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:20.924048 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.925039 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.927695 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:20.928266 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:20.928413 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:20.934411 (+ 5998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:20.934429 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:20.934462 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:20.934591 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:20.938094 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.939090 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.939099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.939102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.939259 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:20.939692 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:20.939702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:20.939705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:20.940165 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:20.940527 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:20.940862 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 01:00:20.941114 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:00:20.942320 4053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:21.043900 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3930
I20260929 01:00:21.061606 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4064
Sep 29 01:00:21 dist-test-slave-k0b7 krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:21.079578 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:21.062627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.3.21.195:33763)
0929 01:00:21.062803 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:21.062808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:21.062827 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:00:21.063077 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:21.063082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:21.063152 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:21.063240 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:21.064731 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:21.065582 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:21.067304 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:21.067729 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:21.067871 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:21.072915 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:21.072933 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:21.072952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:21.073026 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:21.076619 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:21.077416 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:21.077424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:21.077426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:21.077567 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:21.077974 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:21.077978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:21.077980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:21.078333 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:21.078665 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:21.078963 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 01:00:21.079210 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:00:21.080331 4187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:21.189432 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4064
I20260929 01:00:21.209074 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 3703
I20260929 01:00:21.210845 3765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:21.341187 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 3703
2026-09-29T01:00:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4893 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:21 dist-test-slave-k0b7 krb5kdc[4241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:21 dist-test-slave-k0b7 krb5kdc[4241](info): set up 2 sockets
Sep 29 01:00:21 dist-test-slave-k0b7 krb5kdc[4241](info): commencing operation
krb5kdc: starting...
W20260929 01:00:22.519371 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:22 dist-test-slave-k0b7 krb5kdc[4241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:22.760172 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:41903
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:41903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:23.138207 4254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:23.138576 4254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:23.138667 4254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:23.146410 4254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:23.146580 4254 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:00:23.146623 4254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:23.146677 4254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:23.146727 4254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:23.146798 4254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:23.157433 4254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:41903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:41903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:23.158923 4254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:23.160357 4254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:23.173267 4259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:23.173357 4262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:23.173599 4260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:23.174834 4254 server_base.cc:1034] running on GCE node
I20260929 01:00:23.175724 4254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:23.177572 4254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:23.179239 4254 hybrid_clock.cc:648] HybridClock initialized: now 1790643623179085 us; error 76 us; skew 500 ppm
I20260929 01:00:23.180083 4254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:23 dist-test-slave-k0b7 krb5kdc[4241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:23.184617 4254 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:23.186412 4254 webserver.cc:460] Webserver started at http://127.3.21.254:34815/ using document root <none> and password file <none>
I20260929 01:00:23.187273 4254 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:23.187386 4254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:23.187675 4254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:23.191401 4254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "f25ff11ad78e40a59f8f80585a017fac"
format_stamp: "Formatted at 2026-09-29 01:00:23 on dist-test-slave-k0b7"
server_key: "63c7fcd64a4da925bd900f67b08148b6"
server_key_iv: "fdcfe940faec85ff0b0df7429f5d9a8c"
server_key_version: "encryptionkey@0"
I20260929 01:00:23.192291 4254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "f25ff11ad78e40a59f8f80585a017fac"
format_stamp: "Formatted at 2026-09-29 01:00:23 on dist-test-slave-k0b7"
server_key: "63c7fcd64a4da925bd900f67b08148b6"
server_key_iv: "fdcfe940faec85ff0b0df7429f5d9a8c"
server_key_version: "encryptionkey@0"
I20260929 01:00:23.198421 4254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:00:23.202872 4269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:23.205375 4254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:00:23.205650 4254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "f25ff11ad78e40a59f8f80585a017fac"
format_stamp: "Formatted at 2026-09-29 01:00:23 on dist-test-slave-k0b7"
server_key: "63c7fcd64a4da925bd900f67b08148b6"
server_key_iv: "fdcfe940faec85ff0b0df7429f5d9a8c"
server_key_version: "encryptionkey@0"
I20260929 01:00:23.205912 4254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:23.234218 4254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:23.240581 4254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:23.240973 4254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:23.268003 4254 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:41903
I20260929 01:00:23.268023 4330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:41903 every 8 connection(s)
I20260929 01:00:23.269554 4254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:23.274195 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:23.274411 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4254
I20260929 01:00:23.274660 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:23.275557 3159 external_mini_cluster.cc:1428] Setting key 49edd6fc6067830f97ba254d9aab629c
I20260929 01:00:23.291023 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: Bootstrap starting.
Sep 29 01:00:23 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:23.295923 4331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:23.297637 4331 log.cc:824] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:23.304158 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: No bootstrap required, opened a new log
I20260929 01:00:23.310124 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:23.277878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.3.21.254:41903)
0929 01:00:23.278945 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:23.278959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:23.279019 (+ 60us) server_negotiation.cc:407] Connection header received
0929 01:00:23.282205 (+ 3186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:23.282249 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:23.283037 (+ 788us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:23.283601 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:23.285273 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.287589 (+ 2316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.289244 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.290383 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.290631 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:23.295361 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:23.295378 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:23.295405 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:23.295558 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:23.303086 (+ 7528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:23.304329 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:23.304350 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:23.304359 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:23.304682 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:23.305618 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:23.305623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:23.305627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:23.306689 (+ 1062us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:23.307085 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:23.307799 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 01:00:23.308390 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":555,"threads_started":1}
I20260929 01:00:23.315095 4331 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25ff11ad78e40a59f8f80585a017fac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41903 } }
I20260929 01:00:23.315527 4331 raft_consensus.cc:374] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:23.315598 4331 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25ff11ad78e40a59f8f80585a017fac, State: Initialized, Role: FOLLOWER
I20260929 01:00:23.316219 4331 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25ff11ad78e40a59f8f80585a017fac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41903 } }
I20260929 01:00:23.316447 4331 raft_consensus.cc:388] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:23.316608 4331 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:23.316854 4331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:23.319384 4331 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25ff11ad78e40a59f8f80585a017fac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41903 } }
I20260929 01:00:23.319958 4331 leader_election.cc:302] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25ff11ad78e40a59f8f80585a017fac; no voters:
I20260929 01:00:23.320609 4331 leader_election.cc:288] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:23.320959 4336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:23.322391 4336 raft_consensus.cc:686] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [term 1 LEADER]: Becoming Leader. State: Replica: f25ff11ad78e40a59f8f80585a017fac, State: Running, Role: LEADER
I20260929 01:00:23.323055 4331 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:23.322989 4336 consensus_queue.cc:237] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25ff11ad78e40a59f8f80585a017fac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41903 } }
I20260929 01:00:23.326000 4338 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f25ff11ad78e40a59f8f80585a017fac. Latest consensus state: current_term: 1 leader_uuid: "f25ff11ad78e40a59f8f80585a017fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25ff11ad78e40a59f8f80585a017fac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41903 } } }
I20260929 01:00:23.326110 4337 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f25ff11ad78e40a59f8f80585a017fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25ff11ad78e40a59f8f80585a017fac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41903 } } }
I20260929 01:00:23.326409 4338 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:23.326534 4337 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:23.327672 4345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:23.334336 4345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:23.345813 4345 catalog_manager.cc:1347] Generated new cluster ID: 7930268eb83c43e3a2924ad20f9dfa61
I20260929 01:00:23.346035 4345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:23.358676 4345 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:23.360172 4345 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:23.371699 4345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: Generated new TSK 0
I20260929 01:00:23.372876 4345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:23.428042 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:41903
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:23.800307 4357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:23.800652 4357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:23.800777 4357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:23.808605 4357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:23.808735 4357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:23.808830 4357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:00:23.808916 4357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:23.819648 4357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:41903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:23.821110 4357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:23.822469 4357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:23.838616 4365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:23.838649 4363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:23.838672 4362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:23.839103 4357 server_base.cc:1034] running on GCE node
I20260929 01:00:23.840181 4357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:23.841290 4357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:23.842682 4357 hybrid_clock.cc:648] HybridClock initialized: now 1790643623842596 us; error 72 us; skew 500 ppm
I20260929 01:00:23.843037 4357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:23 dist-test-slave-k0b7 krb5kdc[4241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:23.847038 4357 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:23.849308 4357 webserver.cc:460] Webserver started at http://127.3.21.193:36805/ using document root <none> and password file <none>
I20260929 01:00:23.850203 4357 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:23.850317 4357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:23.850754 4357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:23.854171 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "ac209fab712c4400a99064855d91771f"
format_stamp: "Formatted at 2026-09-29 01:00:23 on dist-test-slave-k0b7"
server_key: "9d49e421bc3a320cc5b99121b44c818e"
server_key_iv: "552482a7a53f994fba93bdc645666cdf"
server_key_version: "encryptionkey@0"
I20260929 01:00:23.855175 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "ac209fab712c4400a99064855d91771f"
format_stamp: "Formatted at 2026-09-29 01:00:23 on dist-test-slave-k0b7"
server_key: "9d49e421bc3a320cc5b99121b44c818e"
server_key_iv: "552482a7a53f994fba93bdc645666cdf"
server_key_version: "encryptionkey@0"
I20260929 01:00:23.861079 4357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:00:23.865557 4372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:23.867413 4357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:00:23.867620 4357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "ac209fab712c4400a99064855d91771f"
format_stamp: "Formatted at 2026-09-29 01:00:23 on dist-test-slave-k0b7"
server_key: "9d49e421bc3a320cc5b99121b44c818e"
server_key_iv: "552482a7a53f994fba93bdc645666cdf"
server_key_version: "encryptionkey@0"
I20260929 01:00:23.867880 4357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:23.885251 4357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:23.890435 4357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:23.890797 4357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:23.893649 4357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:23.893884 4357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:23.894052 4357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:23.894126 4357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:00:23 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:23.923575 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:23.894344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:36801 (local address 127.3.21.254:41903)
0929 01:00:23.895055 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:23.895062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:23.896264 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 01:00:23.899142 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:23.899148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:23.899254 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:23.899375 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:23.902107 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.903177 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.906128 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.906816 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.907007 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:23.913414 (+ 6407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:23.913420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:23.913424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:23.913534 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:23.917947 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:23.919281 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:23.919288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:23.919292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:23.919439 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:23.920115 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:23.920123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:23.920128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:23.920479 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:23.920685 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:23.922944 (+ 2259us) server_negotiation.cc:299] Negotiation successful
0929 01:00:23.923250 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":467,"threads_started":1}
I20260929 01:00:23.926076 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:23.895147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.3.21.193:36801)
0929 01:00:23.896065 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:23.896115 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:23.898722 (+ 2607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:23.899723 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:23.899738 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:23.900653 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:23.901861 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:23.901891 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.903362 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.903368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:23.905842 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:23.905869 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.907394 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.907420 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:23.907731 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:23.908631 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:23.908683 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:23.913089 (+ 4406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:23.918211 (+ 5122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:23.918231 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:23.918249 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:23.919082 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:23.919593 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:23.919597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:23.919599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:23.919848 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:23.920716 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:23.920732 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:23.921626 (+ 894us) client_negotiation.cc:769] Sending connection context
0929 01:00:23.922750 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0929 01:00:23.923807 (+ 1057us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":381,"threads_started":1}
I20260929 01:00:23.954635 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:23.933200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:39441 (local address 127.3.21.254:41903)
0929 01:00:23.933467 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:23.933476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:23.934642 (+ 1166us) server_negotiation.cc:407] Connection header received
0929 01:00:23.934902 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:23.934907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:23.934981 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:23.935119 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:23.936160 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.937289 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.939557 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.940399 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.940574 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:23.942941 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:23.942947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:23.942956 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:23.942991 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:23.950706 (+ 7715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:23.951849 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:23.951857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:23.951861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:23.952067 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:23.952611 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:23.952616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:23.952619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:23.953038 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:23.953234 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:23.953968 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 01:00:23.954222 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 01:00:23.954679 4357 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:45681
I20260929 01:00:23.954778 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:23.933115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.3.21.193:39441)
0929 01:00:23.934468 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:23.934518 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:23.934745 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:23.935149 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:23.935154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:23.935685 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:23.935999 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:23.936009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.937478 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.937485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:23.939324 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:23.939352 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:23.940659 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:23.940672 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:23.940924 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:23.941640 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:23.941659 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:23.942714 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:23.950902 (+ 8188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:23.950912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:23.950923 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:23.951623 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:23.952222 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:23.952230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:23.952240 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:23.952466 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:23.953235 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:23.953247 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:23.953588 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 01:00:23.953777 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 01:00:23.954203 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":864,"threads_started":1}
I20260929 01:00:23.954696 4492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:45681 every 8 connection(s)
I20260929 01:00:23.956328 4357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:23.958923 4493 heartbeater.cc:347] Connected to a master server at 127.3.21.254:41903
I20260929 01:00:23.959089 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4357
I20260929 01:00:23.959189 4493 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:23.959329 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:23.959990 3159 external_mini_cluster.cc:1428] Setting key b763ce0b96101826ef93bb0b9e66aba4
I20260929 01:00:23.960115 4493 heartbeater.cc:511] Master 127.3.21.254:41903 requested a full tablet report, sending...
I20260929 01:00:23.962456 4285 ts_manager.cc:194] Registered new tserver with Master: ac209fab712c4400a99064855d91771f (127.3.21.193:45681)
I20260929 01:00:23.964028 4285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:36801
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:24.006964 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:41903
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:24.376466 4497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:24.376832 4497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:24.376936 4497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:24.384833 4497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:24.384985 4497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:24.385075 4497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:00:24.385149 4497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:24.395514 4497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:41903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:24.396981 4497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:24.398311 4497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:24.414225 4505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:24.414366 4502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:24.414333 4503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:24.415184 4497 server_base.cc:1034] running on GCE node
I20260929 01:00:24.415865 4497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:24.416997 4497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:24.418309 4497 hybrid_clock.cc:648] HybridClock initialized: now 1790643624418182 us; error 68 us; skew 500 ppm
I20260929 01:00:24.418854 4497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:24 dist-test-slave-k0b7 krb5kdc[4241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:24.423322 4497 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:24.425567 4497 webserver.cc:460] Webserver started at http://127.3.21.194:46831/ using document root <none> and password file <none>
I20260929 01:00:24.426434 4497 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:24.426673 4497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:24.427047 4497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:24.430665 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "9660440eea534ef5a0f691752a435684"
format_stamp: "Formatted at 2026-09-29 01:00:24 on dist-test-slave-k0b7"
server_key: "3aa91201faa25967f280e98d7cf584fb"
server_key_iv: "f25c86415b5eaee6322a0deb0149c593"
server_key_version: "encryptionkey@0"
I20260929 01:00:24.431613 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "9660440eea534ef5a0f691752a435684"
format_stamp: "Formatted at 2026-09-29 01:00:24 on dist-test-slave-k0b7"
server_key: "3aa91201faa25967f280e98d7cf584fb"
server_key_iv: "f25c86415b5eaee6322a0deb0149c593"
server_key_version: "encryptionkey@0"
I20260929 01:00:24.439788 4497 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 01:00:24.444557 4512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:24.447515 4497 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:00:24.448207 4497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "9660440eea534ef5a0f691752a435684"
format_stamp: "Formatted at 2026-09-29 01:00:24 on dist-test-slave-k0b7"
server_key: "3aa91201faa25967f280e98d7cf584fb"
server_key_iv: "f25c86415b5eaee6322a0deb0149c593"
server_key_version: "encryptionkey@0"
I20260929 01:00:24.448791 4497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:24.482051 4497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:24.487001 4497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:24.487318 4497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:24.490911 4497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:24.491039 4497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:24.491218 4497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:24.491294 4497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:00:24 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:24.521991 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:24.491889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:55943 (local address 127.3.21.254:41903)
0929 01:00:24.492402 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:24.492411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:24.494072 (+ 1661us) server_negotiation.cc:407] Connection header received
0929 01:00:24.497127 (+ 3055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:24.497133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:24.497240 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:24.497385 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:24.500487 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.501373 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.505079 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.505901 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.506012 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:24.512522 (+ 6510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:24.512534 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:24.512542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:24.512609 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:24.517418 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:24.518764 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:24.518772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:24.518779 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:24.518961 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:24.519609 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:24.519615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:24.519618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:24.519907 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:24.520100 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:24.521313 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0929 01:00:24.521583 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260929 01:00:24.523661 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:24.492726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.3.21.194:55943)
0929 01:00:24.493824 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:24.493884 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:24.496555 (+ 2671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:24.497763 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:24.497781 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:24.499027 (+ 1246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:24.500224 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:24.500259 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.501566 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.501576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:24.504730 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:24.504762 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.506471 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.506510 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:24.506952 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:24.508119 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:24.508146 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:24.512168 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:24.517627 (+ 5459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:24.517636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:24.517657 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:24.518476 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:24.519118 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:24.519121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:24.519124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:24.519422 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:24.520221 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:24.520231 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:24.520838 (+ 607us) client_negotiation.cc:769] Sending connection context
0929 01:00:24.521205 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 01:00:24.521843 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":461,"threads_started":1}
I20260929 01:00:24.551908 4497 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:42671
I20260929 01:00:24.551968 4632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:42671 every 8 connection(s)
I20260929 01:00:24.553205 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:24.534396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:43843 (local address 127.3.21.254:41903)
0929 01:00:24.534615 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:24.534623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:24.535549 (+ 926us) server_negotiation.cc:407] Connection header received
0929 01:00:24.535852 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:24.535861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:24.535964 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:24.536120 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:24.537266 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.538642 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.540108 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.540781 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.540943 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:24.544662 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:24.544673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:24.544680 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:24.544731 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:24.549399 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:24.550559 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:24.550567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:24.550573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:24.550762 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:24.551350 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:24.551355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:24.551360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:24.552026 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:24.552218 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:24.552682 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 01:00:24.552930 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:00:24.553511 4597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:24.534440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.3.21.194:43843)
0929 01:00:24.535398 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:24.535425 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:24.535645 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:24.536169 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:24.536179 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:24.536756 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:24.537078 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:24.537089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.538823 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.538829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:24.539933 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:24.539950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:24.541286 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:24.541304 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:24.541520 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:24.542769 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:24.542802 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:24.544343 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:24.549609 (+ 5266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:24.549615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:24.549619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:24.550241 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:24.550925 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:24.550933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:24.550940 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:24.551199 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:24.552226 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:24.552232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:24.552433 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 01:00:24.552597 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 01:00:24.552991 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":547,"threads_started":1}
I20260929 01:00:24.553735 4497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:24.556452 4633 heartbeater.cc:347] Connected to a master server at 127.3.21.254:41903
I20260929 01:00:24.556798 4633 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:24.557818 4633 heartbeater.cc:511] Master 127.3.21.254:41903 requested a full tablet report, sending...
I20260929 01:00:24.559758 4285 ts_manager.cc:194] Registered new tserver with Master: 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671)
I20260929 01:00:24.560611 4285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:55943
I20260929 01:00:24.563680 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4497
I20260929 01:00:24.563855 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:24.564231 3159 external_mini_cluster.cc:1428] Setting key 1083382bd088734dd8aac3a756dfaed1
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:24.611892 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:41903
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:00:24.967628 4493 heartbeater.cc:503] Master 127.3.21.254:41903 was elected leader, sending a full tablet report...
W20260929 01:00:24.979454 4637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:24.979810 4637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:24.979980 4637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:24.988735 4637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:24.988931 4637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:24.989082 4637 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:00:24.989220 4637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:24.999998 4637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:41903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:25.001506 4637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:25.003034 4637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:25.018324 4642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:25.018226 4645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:25.018276 4643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:25.019415 4637 server_base.cc:1034] running on GCE node
I20260929 01:00:25.020143 4637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:25.021373 4637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:25.022678 4637 hybrid_clock.cc:648] HybridClock initialized: now 1790643625022601 us; error 46 us; skew 500 ppm
I20260929 01:00:25.023090 4637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:25 dist-test-slave-k0b7 krb5kdc[4241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:25.027345 4637 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:25.029407 4637 webserver.cc:460] Webserver started at http://127.3.21.195:44905/ using document root <none> and password file <none>
I20260929 01:00:25.030230 4637 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:25.030357 4637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:25.030757 4637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:25.033910 4637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "27b80d08a6774d8c81aedd56796c9c5f"
format_stamp: "Formatted at 2026-09-29 01:00:25 on dist-test-slave-k0b7"
server_key: "8d7cba7b2ea6cd9638e69d534634c483"
server_key_iv: "ede83e35251901748807b8ea5a06caa6"
server_key_version: "encryptionkey@0"
I20260929 01:00:25.034823 4637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "27b80d08a6774d8c81aedd56796c9c5f"
format_stamp: "Formatted at 2026-09-29 01:00:25 on dist-test-slave-k0b7"
server_key: "8d7cba7b2ea6cd9638e69d534634c483"
server_key_iv: "ede83e35251901748807b8ea5a06caa6"
server_key_version: "encryptionkey@0"
I20260929 01:00:25.040822 4637 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:00:25.045275 4652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:25.047206 4637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:25.047449 4637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "27b80d08a6774d8c81aedd56796c9c5f"
format_stamp: "Formatted at 2026-09-29 01:00:25 on dist-test-slave-k0b7"
server_key: "8d7cba7b2ea6cd9638e69d534634c483"
server_key_iv: "ede83e35251901748807b8ea5a06caa6"
server_key_version: "encryptionkey@0"
I20260929 01:00:25.047708 4637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:25.066243 4637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:25.071599 4637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:25.071903 4637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:25.075469 4637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:25.075619 4637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:25.075783 4637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:25.075860 4637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:00:25 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:25.105736 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.075684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:54567 (local address 127.3.21.254:41903)
0929 01:00:25.076255 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.076262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.077526 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 01:00:25.080048 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.080055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.080168 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.080322 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:25.083124 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.084726 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.088300 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.088993 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.089172 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.096405 (+ 7233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:25.096411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:25.096415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:25.096456 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:25.099715 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.101295 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.101305 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.101311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.101527 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.102576 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.102585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.102588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.103115 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:25.103412 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.104804 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.105156 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
I20260929 01:00:25.107873 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.076265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.3.21.195:54567)
0929 01:00:25.077297 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.077372 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.079591 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.080704 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.080722 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.081665 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:25.082796 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.082839 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.084977 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.084984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.088065 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.088094 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.089373 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.089386 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.089844 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.091247 (+ 1403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:25.091301 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:25.096000 (+ 4699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:25.099975 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:25.099993 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:25.100022 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:25.101086 (+ 1064us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:25.101768 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:25.101774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:25.101777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:25.102236 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:25.103319 (+ 1083us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:25.103333 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:25.104057 (+ 724us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.104591 (+ 534us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.105433 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":471,"threads_started":1}
I20260929 01:00:25.136086 4637 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:46227
I20260929 01:00:25.136138 4772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:46227 every 8 connection(s)
I20260929 01:00:25.137903 4637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:25.138226 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.116154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.3.21.195:51435)
0929 01:00:25.116889 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.116912 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.117113 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.117860 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.117870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.118836 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:25.119553 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.119584 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.121396 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.121403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.123117 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.123145 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.124739 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.124746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.124914 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.126073 (+ 1159us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:25.126104 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:25.128368 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:25.133896 (+ 5528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:25.133903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:25.133906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:25.135363 (+ 1457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:25.136045 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:25.136051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:25.136056 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:25.136354 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:25.137069 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:25.137073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:25.137255 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.137466 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.137789 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":400,"threads_started":1}
I20260929 01:00:25.138257 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.115762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:51435 (local address 127.3.21.254:41903)
0929 01:00:25.116037 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.116044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.117019 (+ 975us) server_negotiation.cc:407] Connection header received
0929 01:00:25.117335 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.117343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.117454 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.117554 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:25.119848 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.121132 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.123366 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.124443 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.124658 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.128709 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:25.128720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:25.128726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:25.128786 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:25.133578 (+ 4792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.135620 (+ 2042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.135627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.135631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.135878 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.136496 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.136501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.136504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.136911 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:25.137087 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.137616 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.137899 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:00:25.141124 4773 heartbeater.cc:347] Connected to a master server at 127.3.21.254:41903
I20260929 01:00:25.141410 4773 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:25.142171 4773 heartbeater.cc:511] Master 127.3.21.254:41903 requested a full tablet report, sending...
I20260929 01:00:25.143707 4285 ts_manager.cc:194] Registered new tserver with Master: 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227)
I20260929 01:00:25.144712 4285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:54567
I20260929 01:00:25.145005 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4637
I20260929 01:00:25.145247 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:25.145763 3159 external_mini_cluster.cc:1428] Setting key a7569051048ce7bc12ccb7796c1eeea9
I20260929 01:00:25.149533 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:00:25 dist-test-slave-k0b7 krb5kdc[4241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:00:25 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:25.203444 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.186608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.3.21.254:41903)
0929 01:00:25.186865 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.186873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.187256 (+ 383us) server_negotiation.cc:407] Connection header received
0929 01:00:25.187531 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.187535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.187607 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.187740 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:25.189474 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.190236 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.191716 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.192335 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.192497 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.197395 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:25.197402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:25.197406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:25.197453 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:25.200445 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.201242 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.201249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.201254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.201448 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.201869 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.201874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.201878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.202207 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:25.202481 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.202850 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.203082 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 01:00:25.209218 4285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:25.217111 4285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:25.260450 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.234868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.3.21.193:45681)
0929 01:00:25.248317 (+ 13449us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.248332 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.249577 (+ 1245us) server_negotiation.cc:407] Connection header received
0929 01:00:25.249781 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.249796 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.250153 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.250773 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.252144 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.253851 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.257032 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.259002 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.259197 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.259425 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.259923 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.260109 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13219,"thread_start_us":13057,"threads_started":1}
I20260929 01:00:25.260532 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.234560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45681 (local address 127.0.0.1:49698)
0929 01:00:25.249405 (+ 14845us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.249430 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.249602 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.250753 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.250759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.250800 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.251740 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.251779 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.254008 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.254016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.256702 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.256758 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.259280 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.259288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.259653 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.259694 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.259856 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.260103 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14705,"mutex_wait_us":79,"thread_start_us":11560,"threads_started":1}
I20260929 01:00:25.261179 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.232796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.3.21.194:42671)
0929 01:00:25.248853 (+ 16057us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.248865 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.248903 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:00:25.249027 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.249038 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.249386 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.249622 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.251520 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.253575 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.257166 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.258809 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.258969 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.259232 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.260493 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.260718 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15873,"thread_start_us":15759,"threads_started":1}
I20260929 01:00:25.262153 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.232611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:42671 (local address 127.0.0.1:33368)
0929 01:00:25.243659 (+ 11048us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.243719 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.244100 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.249799 (+ 5699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.249807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.249844 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.251139 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.251162 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.253771 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.253779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.256928 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.256944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.259463 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.259470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.260159 (+ 689us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.260239 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.261493 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.261677 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10530,"mutex_wait_us":149,"thread_start_us":16510,"threads_started":1}
I20260929 01:00:25.263592 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.232896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:46227 (local address 127.0.0.1:47606)
0929 01:00:25.245670 (+ 12774us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.245778 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.245978 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.253400 (+ 7422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.253405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.253443 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.253910 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.253947 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.256601 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.256610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.258974 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.258995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.262763 (+ 3768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.262772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.262985 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.263024 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.263132 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.263290 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12552,"thread_start_us":10081,"threads_started":1}
I20260929 01:00:25.264384 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.233656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.3.21.195:46227)
0929 01:00:25.252347 (+ 18691us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.252358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.252415 (+ 57us) server_negotiation.cc:407] Connection header received
0929 01:00:25.252542 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.252551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.253123 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.253418 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.254353 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.256323 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.259240 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.261747 (+ 2507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.261912 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.262182 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.263867 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.264060 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18248,"thread_start_us":16376,"threads_started":1}
I20260929 01:00:25.265107 4423 tablet_service.cc:1467] Processing CreateTablet for tablet d63d604c546f4aa4b9262f733d7f114e (DEFAULT_TABLE table=test-table [id=7f86ce1eda1245f589e007fda6c6360a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:25.266750 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63d604c546f4aa4b9262f733d7f114e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.268541 4703 tablet_service.cc:1467] Processing CreateTablet for tablet d63d604c546f4aa4b9262f733d7f114e (DEFAULT_TABLE table=test-table [id=7f86ce1eda1245f589e007fda6c6360a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:25.269320 4563 tablet_service.cc:1467] Processing CreateTablet for tablet d63d604c546f4aa4b9262f733d7f114e (DEFAULT_TABLE table=test-table [id=7f86ce1eda1245f589e007fda6c6360a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:25.270146 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63d604c546f4aa4b9262f733d7f114e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.270615 4563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63d604c546f4aa4b9262f733d7f114e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.291900 4792 tablet_bootstrap.cc:492] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Bootstrap starting.
I20260929 01:00:25.293780 4793 tablet_bootstrap.cc:492] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Bootstrap starting.
I20260929 01:00:25.297870 4792 tablet_bootstrap.cc:654] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.298224 4793 tablet_bootstrap.cc:654] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.299741 4792 log.cc:824] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:25.299762 4793 log.cc:824] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:25.299831 4794 tablet_bootstrap.cc:492] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Bootstrap starting.
I20260929 01:00:25.304365 4794 tablet_bootstrap.cc:654] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.305606 4794 log.cc:824] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:25.309226 4792 tablet_bootstrap.cc:492] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: No bootstrap required, opened a new log
I20260929 01:00:25.309288 4794 tablet_bootstrap.cc:492] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: No bootstrap required, opened a new log
I20260929 01:00:25.309288 4793 tablet_bootstrap.cc:492] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: No bootstrap required, opened a new log
I20260929 01:00:25.309587 4792 ts_tablet_manager.cc:1397] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260929 01:00:25.309594 4793 ts_tablet_manager.cc:1397] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260929 01:00:25.309594 4794 ts_tablet_manager.cc:1397] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 01:00:25.321285 4792 raft_consensus.cc:348] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.321821 4792 raft_consensus.cc:374] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.321954 4792 raft_consensus.cc:729] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac209fab712c4400a99064855d91771f, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.322772 4792 consensus_queue.cc:260] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.323374 4793 raft_consensus.cc:348] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.323392 4794 raft_consensus.cc:348] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.323909 4793 raft_consensus.cc:374] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.323916 4794 raft_consensus.cc:374] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.324038 4794 raft_consensus.cc:729] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b80d08a6774d8c81aedd56796c9c5f, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.324038 4793 raft_consensus.cc:729] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9660440eea534ef5a0f691752a435684, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.324921 4793 consensus_queue.cc:260] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.325330 4792 ts_tablet_manager.cc:1428] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 01:00:25.325794 4794 consensus_queue.cc:260] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.327050 4633 heartbeater.cc:503] Master 127.3.21.254:41903 was elected leader, sending a full tablet report...
I20260929 01:00:25.327996 4793 ts_tablet_manager.cc:1428] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Time spent starting tablet: real 0.018s user 0.019s sys 0.001s
I20260929 01:00:25.329455 4773 heartbeater.cc:503] Master 127.3.21.254:41903 was elected leader, sending a full tablet report...
I20260929 01:00:25.330663 4794 ts_tablet_manager.cc:1428] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20260929 01:00:25.355379 4798 raft_consensus.cc:482] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:25.355912 4798 raft_consensus.cc:504] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.357736 4798 leader_election.cc:288] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671), 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227)
I20260929 01:00:25.370658 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.358411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:57805 (local address 127.3.21.194:42671)
0929 01:00:25.358616 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.359063 (+ 447us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.360323 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 01:00:25.360409 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.360414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.360505 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.360625 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.361864 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.363139 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.367611 (+ 4472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.369172 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.369336 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.369419 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.370151 (+ 732us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.370311 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:00:25.370651 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.358202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:57115 (local address 127.3.21.195:46227)
0929 01:00:25.358431 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.358438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.360120 (+ 1682us) server_negotiation.cc:407] Connection header received
0929 01:00:25.360394 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.360400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.360517 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.360625 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.361902 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.363109 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.367602 (+ 4493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.369057 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.369335 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.369419 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.370181 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.370322 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:00:25.370929 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.358197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:42671 (local address 127.3.21.193:57805)
0929 01:00:25.360106 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.360129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.360250 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.360690 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.360696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.360801 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.361426 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.361447 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.363425 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.363432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.367284 (+ 3852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.367317 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.369533 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.369543 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.369834 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.369914 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.370043 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.370235 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"mutex_wait_us":478,"spinlock_wait_cycles":31104,"thread_start_us":981,"threads_started":1}
I20260929 01:00:25.371379 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.358143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:46227 (local address 127.3.21.193:57115)
0929 01:00:25.359918 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.359944 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.360184 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.360690 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.360696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.360728 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.361466 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.361478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.363425 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.363432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.367286 (+ 3854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.367316 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.369542 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.369548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.369824 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.369917 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.370043 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.370235 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":477,"thread_start_us":1325,"threads_started":1}
I20260929 01:00:25.372852 4723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d604c546f4aa4b9262f733d7f114e" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b80d08a6774d8c81aedd56796c9c5f" is_pre_election: true
I20260929 01:00:25.372684 4587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d604c546f4aa4b9262f733d7f114e" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9660440eea534ef5a0f691752a435684" is_pre_election: true
I20260929 01:00:25.373646 4723 raft_consensus.cc:2454] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 0.
I20260929 01:00:25.373620 4587 raft_consensus.cc:2454] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 0.
I20260929 01:00:25.374792 4373 leader_election.cc:302] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9660440eea534ef5a0f691752a435684, ac209fab712c4400a99064855d91771f; no voters:
I20260929 01:00:25.375452 4798 raft_consensus.cc:2781] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:25.375634 4798 raft_consensus.cc:482] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:25.375720 4798 raft_consensus.cc:3037] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.379055 4798 raft_consensus.cc:504] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
I20260929 01:00:25.379761 4798 leader_election.cc:288] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 election: Requested vote from peers 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671), 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227)
I20260929 01:00:25.380417 4587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d604c546f4aa4b9262f733d7f114e" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9660440eea534ef5a0f691752a435684"
I20260929 01:00:25.380450 4723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63d604c546f4aa4b9262f733d7f114e" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b80d08a6774d8c81aedd56796c9c5f"
I20260929 01:00:25.380667 4587 raft_consensus.cc:3037] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.380666 4723 raft_consensus.cc:3037] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.383893 4723 raft_consensus.cc:2454] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 1.
I20260929 01:00:25.384054 4587 raft_consensus.cc:2454] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 1.
I20260929 01:00:25.384788 4376 leader_election.cc:302] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b80d08a6774d8c81aedd56796c9c5f, ac209fab712c4400a99064855d91771f; no voters:
I20260929 01:00:25.386132 4798 raft_consensus.cc:2781] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:25.386942 4798 raft_consensus.cc:686] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 1 LEADER]: Becoming Leader. State: Replica: ac209fab712c4400a99064855d91771f, State: Running, Role: LEADER
I20260929 01:00:25.387565 4798 consensus_queue.cc:237] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } }
W20260929 01:00:25.390534 4774 tablet.cc:2369] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:25.393616 4286 catalog_manager.cc:5495] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac209fab712c4400a99064855d91771f (127.3.21.193). New cstate: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } health_report { overall_health: HEALTHY } } }
I20260929 01:00:25.435545 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.428490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.3.21.193:45681)
0929 01:00:25.428736 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.428743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.428972 (+ 229us) server_negotiation.cc:407] Connection header received
0929 01:00:25.429302 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.429307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.429432 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.429598 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:25.430016 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.431008 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.432546 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.433022 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.433186 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.434065 (+ 879us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.434832 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.435031 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.435158 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.435254 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:00:25.448716 4587 raft_consensus.cc:1261] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Refusing update from remote peer ac209fab712c4400a99064855d91771f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:25.448956 4723 raft_consensus.cc:1261] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Refusing update from remote peer ac209fab712c4400a99064855d91771f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:25.450120 4803 consensus_queue.cc:1035] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:25.450636 4798 consensus_queue.cc:1035] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:00:25.459389 4494 tablet.cc:2369] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:25.489164 4285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:25.489552 4285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:25.494335 4285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: Sending DeleteTablet for 3 replicas of tablet d63d604c546f4aa4b9262f733d7f114e
I20260929 01:00:25.496333 4563 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d604c546f4aa4b9262f733d7f114e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:25 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33368
I20260929 01:00:25.496382 4703 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d604c546f4aa4b9262f733d7f114e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:25 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:47606
I20260929 01:00:25.497677 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet d63d604c546f4aa4b9262f733d7f114e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:25 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49698
I20260929 01:00:25.500550 4818 tablet_replica.cc:330] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: stopping tablet replica
I20260929 01:00:25.501394 4818 raft_consensus.cc:2229] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:25.501277 4285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:25.502091 4285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:25.502837 4818 raft_consensus.cc:2258] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:25.503959 4816 tablet_replica.cc:330] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: stopping tablet replica
I20260929 01:00:25.504612 4817 tablet_replica.cc:330] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: stopping tablet replica
I20260929 01:00:25.505172 4817 raft_consensus.cc:2229] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:25.505713 4818 ts_tablet_manager.cc:1905] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:25.505688 4816 raft_consensus.cc:2229] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:25.506376 4817 raft_consensus.cc:2258] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:25.506985 4816 raft_consensus.cc:2258] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:25.509631 4817 ts_tablet_manager.cc:1905] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:25.511320 4816 ts_tablet_manager.cc:1905] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:25.512274 4703 tablet_service.cc:1467] Processing CreateTablet for tablet 6803758d640f4d16a3607b9cd4a8433d (DEFAULT_TABLE table=test-table [id=ea41f07c7f98407494fdce67039ab958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:25.512467 4563 tablet_service.cc:1467] Processing CreateTablet for tablet 6803758d640f4d16a3607b9cd4a8433d (DEFAULT_TABLE table=test-table [id=ea41f07c7f98407494fdce67039ab958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:25.512806 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6803758d640f4d16a3607b9cd4a8433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.512953 4563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6803758d640f4d16a3607b9cd4a8433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.518688 4817 ts_tablet_manager.cc:1918] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:25.519037 4817 log.cc:1196] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/d63d604c546f4aa4b9262f733d7f114e
I20260929 01:00:25.519749 4817 ts_tablet_manager.cc:1939] T d63d604c546f4aa4b9262f733d7f114e P 9660440eea534ef5a0f691752a435684: Deleting consensus metadata
I20260929 01:00:25.520198 4816 ts_tablet_manager.cc:1918] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:25.520398 4816 log.cc:1196] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/d63d604c546f4aa4b9262f733d7f114e
I20260929 01:00:25.520864 4794 tablet_bootstrap.cc:492] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Bootstrap starting.
I20260929 01:00:25.521032 4816 ts_tablet_manager.cc:1939] T d63d604c546f4aa4b9262f733d7f114e P 27b80d08a6774d8c81aedd56796c9c5f: Deleting consensus metadata
I20260929 01:00:25.523049 4794 tablet_bootstrap.cc:654] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.523049 4270 catalog_manager.cc:4841] TS 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671): tablet d63d604c546f4aa4b9262f733d7f114e (table test-table [id=7f86ce1eda1245f589e007fda6c6360a]) successfully deleted
I20260929 01:00:25.523468 4273 catalog_manager.cc:4841] TS 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227): tablet d63d604c546f4aa4b9262f733d7f114e (table test-table [id=7f86ce1eda1245f589e007fda6c6360a]) successfully deleted
I20260929 01:00:25.524164 4423 tablet_service.cc:1467] Processing CreateTablet for tablet 6803758d640f4d16a3607b9cd4a8433d (DEFAULT_TABLE table=test-table [id=ea41f07c7f98407494fdce67039ab958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:25.524770 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6803758d640f4d16a3607b9cd4a8433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.527324 4793 tablet_bootstrap.cc:492] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Bootstrap starting.
I20260929 01:00:25.529409 4793 tablet_bootstrap.cc:654] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.536422 4792 tablet_bootstrap.cc:492] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Bootstrap starting.
I20260929 01:00:25.537879 4818 ts_tablet_manager.cc:1918] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:25.538096 4818 log.cc:1196] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/d63d604c546f4aa4b9262f733d7f114e
I20260929 01:00:25.538564 4792 tablet_bootstrap.cc:654] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.539261 4818 ts_tablet_manager.cc:1939] T d63d604c546f4aa4b9262f733d7f114e P ac209fab712c4400a99064855d91771f: Deleting consensus metadata
I20260929 01:00:25.540141 4794 tablet_bootstrap.cc:492] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: No bootstrap required, opened a new log
I20260929 01:00:25.540333 4794 ts_tablet_manager.cc:1397] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.012s
I20260929 01:00:25.541247 4794 raft_consensus.cc:348] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.541519 4794 raft_consensus.cc:374] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.541628 4794 raft_consensus.cc:729] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b80d08a6774d8c81aedd56796c9c5f, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.541857 4272 catalog_manager.cc:4841] TS ac209fab712c4400a99064855d91771f (127.3.21.193:45681): tablet d63d604c546f4aa4b9262f733d7f114e (table test-table [id=7f86ce1eda1245f589e007fda6c6360a]) successfully deleted
I20260929 01:00:25.541961 4794 consensus_queue.cc:260] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.543041 4793 tablet_bootstrap.cc:492] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: No bootstrap required, opened a new log
I20260929 01:00:25.543236 4793 ts_tablet_manager.cc:1397] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260929 01:00:25.545775 4794 ts_tablet_manager.cc:1428] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260929 01:00:25.546037 4793 raft_consensus.cc:348] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.546365 4793 raft_consensus.cc:374] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.546525 4793 raft_consensus.cc:729] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9660440eea534ef5a0f691752a435684, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.546901 4793 consensus_queue.cc:260] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.548914 4793 ts_tablet_manager.cc:1428] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260929 01:00:25.550851 4792 tablet_bootstrap.cc:492] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: No bootstrap required, opened a new log
I20260929 01:00:25.551131 4792 ts_tablet_manager.cc:1397] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20260929 01:00:25.552121 4792 raft_consensus.cc:348] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.552397 4792 raft_consensus.cc:374] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.552534 4792 raft_consensus.cc:729] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac209fab712c4400a99064855d91771f, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.552850 4792 consensus_queue.cc:260] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.553745 4792 ts_tablet_manager.cc:1428] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260929 01:00:25.557080 4634 tablet.cc:2369] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:25.637383 4800 raft_consensus.cc:482] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:25.637657 4800 raft_consensus.cc:504] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.639583 4800 leader_election.cc:288] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac209fab712c4400a99064855d91771f (127.3.21.193:45681), 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671)
I20260929 01:00:25.647694 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.640154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:35123 (local address 127.3.21.194:42671)
0929 01:00:25.640381 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.640388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.640924 (+ 536us) server_negotiation.cc:407] Connection header received
0929 01:00:25.641277 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.641284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.641380 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.641468 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.642156 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.643133 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.645348 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.646671 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.646878 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.646967 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.647309 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.647425 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 01:00:25.648147 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.639842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45681 (local address 127.3.21.195:57417)
0929 01:00:25.641448 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.641467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.641600 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.642037 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.642044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.642071 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.642404 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.642415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.644029 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.644035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.645780 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.645791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.647253 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.647260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.647464 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.647499 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.647596 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.647769 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"mutex_wait_us":73,"thread_start_us":439,"threads_started":1}
I20260929 01:00:25.648485 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.639841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:42671 (local address 127.3.21.195:35123)
0929 01:00:25.640573 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.640604 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.641089 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.641708 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.641712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.641736 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:25.641978 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.641993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.643353 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.643358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.645161 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.645178 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.646889 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.646896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.647084 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.647128 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.647989 (+ 861us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.648154 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":1252,"threads_started":1}
I20260929 01:00:25.648553 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.640186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:57417 (local address 127.3.21.193:45681)
0929 01:00:25.640366 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.640373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.641750 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 01:00:25.641828 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.641834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.641907 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.641998 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:25.642656 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.643878 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.646009 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.647016 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.647176 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.647274 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.647640 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.648039 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:00:25.649158 4587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6803758d640f4d16a3607b9cd4a8433d" candidate_uuid: "27b80d08a6774d8c81aedd56796c9c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9660440eea534ef5a0f691752a435684" is_pre_election: true
I20260929 01:00:25.649446 4587 raft_consensus.cc:2454] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b80d08a6774d8c81aedd56796c9c5f in term 0.
I20260929 01:00:25.649300 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6803758d640f4d16a3607b9cd4a8433d" candidate_uuid: "27b80d08a6774d8c81aedd56796c9c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac209fab712c4400a99064855d91771f" is_pre_election: true
I20260929 01:00:25.649850 4443 raft_consensus.cc:2454] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b80d08a6774d8c81aedd56796c9c5f in term 0.
I20260929 01:00:25.650002 4653 leader_election.cc:302] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b80d08a6774d8c81aedd56796c9c5f, 9660440eea534ef5a0f691752a435684; no voters:
I20260929 01:00:25.650451 4800 raft_consensus.cc:2781] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:25.650657 4800 raft_consensus.cc:482] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:25.650782 4800 raft_consensus.cc:3037] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.653906 4800 raft_consensus.cc:504] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.654678 4800 leader_election.cc:288] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [CANDIDATE]: Term 1 election: Requested vote from peers ac209fab712c4400a99064855d91771f (127.3.21.193:45681), 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671)
I20260929 01:00:25.655194 4443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6803758d640f4d16a3607b9cd4a8433d" candidate_uuid: "27b80d08a6774d8c81aedd56796c9c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac209fab712c4400a99064855d91771f"
I20260929 01:00:25.655260 4587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6803758d640f4d16a3607b9cd4a8433d" candidate_uuid: "27b80d08a6774d8c81aedd56796c9c5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9660440eea534ef5a0f691752a435684"
I20260929 01:00:25.655489 4587 raft_consensus.cc:3037] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.655484 4443 raft_consensus.cc:3037] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.658245 4443 raft_consensus.cc:2454] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b80d08a6774d8c81aedd56796c9c5f in term 1.
I20260929 01:00:25.658721 4587 raft_consensus.cc:2454] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b80d08a6774d8c81aedd56796c9c5f in term 1.
I20260929 01:00:25.658845 4655 leader_election.cc:302] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b80d08a6774d8c81aedd56796c9c5f, ac209fab712c4400a99064855d91771f; no voters:
I20260929 01:00:25.659197 4800 raft_consensus.cc:2781] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:25.659680 4800 raft_consensus.cc:686] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 1 LEADER]: Becoming Leader. State: Replica: 27b80d08a6774d8c81aedd56796c9c5f, State: Running, Role: LEADER
I20260929 01:00:25.660151 4800 consensus_queue.cc:237] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.664497 4286 catalog_manager.cc:5495] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195). New cstate: current_term: 1 leader_uuid: "27b80d08a6774d8c81aedd56796c9c5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } health_report { overall_health: HEALTHY } } }
Sep 29 01:00:25 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:25.731689 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.709057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41903 (local address 127.0.0.1:46302)
0929 01:00:25.709767 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.710196 (+ 429us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.710569 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.711374 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.711384 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.712441 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:25.712814 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.712831 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.714295 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.714301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.716110 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.716132 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.716955 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.716967 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.717155 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.718048 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:25.718077 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:25.721541 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:25.726898 (+ 5357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:25.726921 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:25.726936 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:25.728192 (+ 1256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:25.728770 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:25.728775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:25.728779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:25.728950 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:25.730014 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:25.730026 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:25.730263 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.730464 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.731251 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":355,"threads_started":1}
I20260929 01:00:25.731984 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.709147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.3.21.254:41903)
0929 01:00:25.710982 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.710990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.711014 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:00:25.711074 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.711079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.711160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.711281 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:25.713096 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.714105 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.716311 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.716738 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.716860 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.721814 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:25.721823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:25.721828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:25.721871 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:25.726322 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.728390 (+ 2068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.728396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.728400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.728592 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:25.729248 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:25.729255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:25.729257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:25.729711 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:25.729990 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.730793 (+ 803us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.731520 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":348,"server-negotiator.queue_time_us":1692,"thread_start_us":1897,"threads_started":1}
I20260929 01:00:25.737351 4286 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:46302:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:00:25.738468 4286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:25.746729 4563 tablet_service.cc:1467] Processing CreateTablet for tablet aa35db99a9da48d4b4d9f00e4db179dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f202e0e4976647999ea0ee0cd871c72f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:00:25.746738 4423 tablet_service.cc:1467] Processing CreateTablet for tablet aa35db99a9da48d4b4d9f00e4db179dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f202e0e4976647999ea0ee0cd871c72f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:00:25.747073 4703 tablet_service.cc:1467] Processing CreateTablet for tablet aa35db99a9da48d4b4d9f00e4db179dc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f202e0e4976647999ea0ee0cd871c72f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:00:25.747413 4563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa35db99a9da48d4b4d9f00e4db179dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.747428 4423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa35db99a9da48d4b4d9f00e4db179dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.747650 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa35db99a9da48d4b4d9f00e4db179dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:25.755687 4793 tablet_bootstrap.cc:492] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: Bootstrap starting.
I20260929 01:00:25.757323 4793 tablet_bootstrap.cc:654] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.758739 4794 tablet_bootstrap.cc:492] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: Bootstrap starting.
I20260929 01:00:25.759239 4792 tablet_bootstrap.cc:492] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: Bootstrap starting.
I20260929 01:00:25.761173 4792 tablet_bootstrap.cc:654] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.761194 4794 tablet_bootstrap.cc:654] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:25.772292 4794 tablet_bootstrap.cc:492] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: No bootstrap required, opened a new log
I20260929 01:00:25.772418 4793 tablet_bootstrap.cc:492] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: No bootstrap required, opened a new log
I20260929 01:00:25.772606 4793 ts_tablet_manager.cc:1397] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260929 01:00:25.772720 4792 tablet_bootstrap.cc:492] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: No bootstrap required, opened a new log
I20260929 01:00:25.772781 4794 ts_tablet_manager.cc:1397] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 01:00:25.772953 4792 ts_tablet_manager.cc:1397] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260929 01:00:25.773701 4794 raft_consensus.cc:348] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.773736 4792 raft_consensus.cc:348] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.773849 4793 raft_consensus.cc:348] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.774022 4792 raft_consensus.cc:374] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.774096 4793 raft_consensus.cc:374] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.774116 4792 raft_consensus.cc:729] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac209fab712c4400a99064855d91771f, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.774186 4793 raft_consensus.cc:729] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9660440eea534ef5a0f691752a435684, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.774154 4794 raft_consensus.cc:374] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:25.774389 4794 raft_consensus.cc:729] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b80d08a6774d8c81aedd56796c9c5f, State: Initialized, Role: FOLLOWER
I20260929 01:00:25.774390 4792 consensus_queue.cc:260] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.774565 4793 consensus_queue.cc:260] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.775017 4799 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.775326 4792 ts_tablet_manager.cc:1428] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:25.775039 4811 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.775372 4799 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.775460 4811 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.776288 4794 consensus_queue.cc:260] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.776500 4793 ts_tablet_manager.cc:1428] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 01:00:25.776826 4800 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.777146 4800 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.777689 4794 ts_tablet_manager.cc:1428] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 01:00:25.798743 4811 raft_consensus.cc:482] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:25.799019 4811 raft_consensus.cc:504] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.799880 4811 leader_election.cc:288] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671), 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227)
I20260929 01:00:25.800460 4723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35db99a9da48d4b4d9f00e4db179dc" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b80d08a6774d8c81aedd56796c9c5f" is_pre_election: true
I20260929 01:00:25.800451 4587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35db99a9da48d4b4d9f00e4db179dc" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9660440eea534ef5a0f691752a435684" is_pre_election: true
I20260929 01:00:25.800709 4723 raft_consensus.cc:2454] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 0.
I20260929 01:00:25.800729 4587 raft_consensus.cc:2454] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 0.
I20260929 01:00:25.801180 4376 leader_election.cc:302] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b80d08a6774d8c81aedd56796c9c5f, ac209fab712c4400a99064855d91771f; no voters:
I20260929 01:00:25.801498 4811 raft_consensus.cc:2781] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:25.801640 4811 raft_consensus.cc:482] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:25.801743 4811 raft_consensus.cc:3037] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.805243 4811 raft_consensus.cc:504] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.806253 4811 leader_election.cc:288] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 election: Requested vote from peers 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671), 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227)
I20260929 01:00:25.807327 4723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35db99a9da48d4b4d9f00e4db179dc" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b80d08a6774d8c81aedd56796c9c5f"
I20260929 01:00:25.807379 4587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa35db99a9da48d4b4d9f00e4db179dc" candidate_uuid: "ac209fab712c4400a99064855d91771f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9660440eea534ef5a0f691752a435684"
I20260929 01:00:25.807634 4723 raft_consensus.cc:3037] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.807708 4587 raft_consensus.cc:3037] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:25.811165 4723 raft_consensus.cc:2454] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 1.
I20260929 01:00:25.811167 4587 raft_consensus.cc:2454] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac209fab712c4400a99064855d91771f in term 1.
I20260929 01:00:25.811914 4373 leader_election.cc:302] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27b80d08a6774d8c81aedd56796c9c5f, 9660440eea534ef5a0f691752a435684, ac209fab712c4400a99064855d91771f; no voters:
I20260929 01:00:25.812311 4811 raft_consensus.cc:2781] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:25.812525 4811 raft_consensus.cc:686] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [term 1 LEADER]: Becoming Leader. State: Replica: ac209fab712c4400a99064855d91771f, State: Running, Role: LEADER
I20260929 01:00:25.812836 4811 consensus_queue.cc:237] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } }
I20260929 01:00:25.815210 4798 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: TxnStatusTablet state changed. Reason: New leader ac209fab712c4400a99064855d91771f. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.815665 4798 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:00:25.816704 4837 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:00:25.817023 4837 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:00:25.817103 4286 catalog_manager.cc:5495] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f reported cstate change: term changed from 0 to 1, leader changed from <none> to ac209fab712c4400a99064855d91771f (127.3.21.193). New cstate: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:25.853884 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.846682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.3.21.193:45681)
0929 01:00:25.846845 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.846851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.846947 (+ 96us) server_negotiation.cc:407] Connection header received
0929 01:00:25.847350 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.847357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.847427 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.847532 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:25.848002 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.848913 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.851500 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.852209 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.852330 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.853007 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.853126 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.853209 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.853420 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.853521 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 01:00:25.853901 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.846583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45681 (local address 127.0.0.1:49722)
0929 01:00:25.846842 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.846865 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.847030 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.847607 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.847613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.847628 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:25.847832 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.847844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.849234 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.849241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.851303 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.851323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.852413 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.852422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.852610 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.852697 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.853238 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.853259 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.853368 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.853552 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:00:25.858299 4587 raft_consensus.cc:1261] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Refusing update from remote peer ac209fab712c4400a99064855d91771f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:25.858273 4723 raft_consensus.cc:1261] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Refusing update from remote peer ac209fab712c4400a99064855d91771f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:25.859031 4811 consensus_queue.cc:1035] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:25.859395 4798 consensus_queue.cc:1035] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:25.863682 4800 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: TxnStatusTablet state changed. Reason: New leader ac209fab712c4400a99064855d91771f. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.864054 4800 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.866711 4806 mvcc.cc:205] Tried to move back new op lower bound from 7334476291508310016 to 7334476291332222976. Current Snapshot: MvccSnapshot[applied={T|T < 7334476291508310016}]
I20260929 01:00:25.868806 4798 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.869302 4798 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:00:25.872431 4800 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.872853 4800 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P 27b80d08a6774d8c81aedd56796c9c5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.873822 4799 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: TxnStatusTablet state changed. Reason: New leader ac209fab712c4400a99064855d91771f. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.874172 4799 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.879012 4798 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.879355 4798 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P ac209fab712c4400a99064855d91771f: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:00:25.882751 4799 tablet_replica.cc:439] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac209fab712c4400a99064855d91771f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 } } peers { permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 } } peers { permanent_uuid: "27b80d08a6774d8c81aedd56796c9c5f" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46227 } } }
I20260929 01:00:25.883091 4799 tablet_replica.cc:442] T aa35db99a9da48d4b4d9f00e4db179dc P 9660440eea534ef5a0f691752a435684: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:25.888443 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.880652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.3.21.195:46227)
0929 01:00:25.881275 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.881282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.881305 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:00:25.881419 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.881425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.881548 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.882272 (+ 724us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:25.882452 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.883734 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.885579 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.886051 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.886165 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.886895 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.887509 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.887633 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.887856 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.887965 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":371,"threads_started":1}
I20260929 01:00:25.909399 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.902455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:49425 (local address 127.3.21.193:45681)
0929 01:00:25.902872 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.902879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.903017 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:00:25.903311 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.903319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.903440 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.903546 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:25.904112 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.905024 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.906845 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.907388 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.907538 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.908032 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.908633 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.908765 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.908925 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.909022 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 01:00:25.909484 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.902181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45681 (local address 127.3.21.195:49425)
0929 01:00:25.902874 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.902919 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.903119 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.903589 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.903595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.903607 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:25.903950 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.903959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.905176 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.905183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.906657 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.906673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.907606 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.907613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.907770 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.907838 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.908770 (+ 932us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.908785 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.908894 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.909057 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":334,"threads_started":1}
I20260929 01:00:25.920782 4587 raft_consensus.cc:1261] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Refusing update from remote peer 27b80d08a6774d8c81aedd56796c9c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:25.921305 4443 raft_consensus.cc:1261] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Refusing update from remote peer 27b80d08a6774d8c81aedd56796c9c5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:25.921988 4824 consensus_queue.cc:1035] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9660440eea534ef5a0f691752a435684" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:25.922408 4800 consensus_queue.cc:1035] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac209fab712c4400a99064855d91771f" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:25.977773 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.969475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:44411 (local address 127.3.21.195:46227)
0929 01:00:25.969659 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:25.969666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:25.971311 (+ 1645us) server_negotiation.cc:407] Connection header received
0929 01:00:25.971529 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:25.971539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:25.971603 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:25.971720 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:25.972618 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.973651 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.975891 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.976325 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.976456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.976923 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.977072 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.977155 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:25.977386 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 01:00:25.977475 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:00:25.979298 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:25.968992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:46227 (local address 127.3.21.193:44411)
0929 01:00:25.971195 (+ 2203us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:25.971215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:25.971383 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:25.971967 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:25.971972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:25.971988 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:25.972411 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.972426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.974207 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.974215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.975739 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:25.975760 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:25.976545 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:25.976556 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:25.976675 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:25.976736 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:25.977207 (+ 471us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:25.977219 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:00:25.978309 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0929 01:00:25.978863 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2011,"thread_start_us":1734,"threads_started":1}
I20260929 01:00:26.034844 4284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:26.035097 4284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:26.038621 4284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f25ff11ad78e40a59f8f80585a017fac: Sending DeleteTablet for 3 replicas of tablet 6803758d640f4d16a3607b9cd4a8433d
I20260929 01:00:26.039578 4563 tablet_service.cc:1514] Processing DeleteTablet for tablet 6803758d640f4d16a3607b9cd4a8433d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:26 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33368
I20260929 01:00:26.040433 4703 tablet_service.cc:1514] Processing DeleteTablet for tablet 6803758d640f4d16a3607b9cd4a8433d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:26 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:47606
I20260929 01:00:26.041065 4423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6803758d640f4d16a3607b9cd4a8433d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:26 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49698
I20260929 01:00:26.041455 4818 tablet_replica.cc:330] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: stopping tablet replica
I20260929 01:00:26.041745 4818 raft_consensus.cc:2229] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:26.041996 4818 raft_consensus.cc:2258] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:26.043910 4818 ts_tablet_manager.cc:1905] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:26.055244 4852 tablet_replica.cc:330] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: stopping tablet replica
I20260929 01:00:26.055642 4852 raft_consensus.cc:2229] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:26.056005 4852 raft_consensus.cc:2258] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:26.057305 4852 ts_tablet_manager.cc:1905] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:26.061808 4818 ts_tablet_manager.cc:1918] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:26.065923 4852 ts_tablet_manager.cc:1918] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:26.066102 4852 log.cc:1196] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/6803758d640f4d16a3607b9cd4a8433d
I20260929 01:00:26.067435 4852 ts_tablet_manager.cc:1939] T 6803758d640f4d16a3607b9cd4a8433d P 9660440eea534ef5a0f691752a435684: Deleting consensus metadata
I20260929 01:00:26.068190 4818 log.cc:1196] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/6803758d640f4d16a3607b9cd4a8433d
I20260929 01:00:26.068820 4270 catalog_manager.cc:4841] TS 9660440eea534ef5a0f691752a435684 (127.3.21.194:42671): tablet 6803758d640f4d16a3607b9cd4a8433d (table test-table [id=ea41f07c7f98407494fdce67039ab958]) successfully deleted
I20260929 01:00:26.069423 4818 ts_tablet_manager.cc:1939] T 6803758d640f4d16a3607b9cd4a8433d P ac209fab712c4400a99064855d91771f: Deleting consensus metadata
I20260929 01:00:26.070829 4272 catalog_manager.cc:4841] TS ac209fab712c4400a99064855d91771f (127.3.21.193:45681): tablet 6803758d640f4d16a3607b9cd4a8433d (table test-table [id=ea41f07c7f98407494fdce67039ab958]) successfully deleted
I20260929 01:00:26.071973 4853 tablet_replica.cc:330] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: stopping tablet replica
I20260929 01:00:26.072674 4853 raft_consensus.cc:2229] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:26.073443 4853 raft_consensus.cc:2258] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:26.074805 4853 ts_tablet_manager.cc:1905] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 01:00:26 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:26.084070 4853 ts_tablet_manager.cc:1918] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:26.084370 4853 log.cc:1196] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/6803758d640f4d16a3607b9cd4a8433d
I20260929 01:00:26.085011 4853 ts_tablet_manager.cc:1939] T 6803758d640f4d16a3607b9cd4a8433d P 27b80d08a6774d8c81aedd56796c9c5f: Deleting consensus metadata
I20260929 01:00:26.086342 4273 catalog_manager.cc:4841] TS 27b80d08a6774d8c81aedd56796c9c5f (127.3.21.195:46227): tablet 6803758d640f4d16a3607b9cd4a8433d (table test-table [id=ea41f07c7f98407494fdce67039ab958]) successfully deleted
I20260929 01:00:26.090126 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.058179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.3.21.193:45681)
0929 01:00:26.058407 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:26.058414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:26.067177 (+ 8763us) server_negotiation.cc:407] Connection header received
0929 01:00:26.069451 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:26.069456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:26.069546 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:26.069651 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:26.072212 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.073613 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.075637 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.076342 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.076452 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:26.082114 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:26.082129 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:26.082149 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:26.082204 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:26.087007 (+ 4803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.087852 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.087863 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.087866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.088010 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.088409 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.088414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.088417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.088832 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:26.089177 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:26.089490 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 01:00:26.089725 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 01:00:26.091024 4488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49730
I20260929 01:00:26.115890 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.095075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.3.21.254:41903)
0929 01:00:26.095323 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:26.095333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:26.101557 (+ 6224us) server_negotiation.cc:407] Connection header received
0929 01:00:26.101858 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:26.101863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:26.101944 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:26.102010 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:26.103560 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.104405 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.105694 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.106227 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.106399 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:26.109538 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:26.109544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:26.109547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:26.109590 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:26.113006 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.113772 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.113778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.113781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.113920 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.114307 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.114311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.114315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.114693 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:26.114983 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:26.115323 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 01:00:26.115572 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 01:00:26.116446 4284 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46312
I20260929 01:00:26.119899 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4357
I20260929 01:00:26.140929 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.121219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.3.21.193:45681)
0929 01:00:26.121393 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:26.121400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:26.121420 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:00:26.121612 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:26.121618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:26.121721 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:26.121786 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:26.123300 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.124427 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.126077 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.126899 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.127042 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:26.130178 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:26.130183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:26.130186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:26.130270 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:26.135594 (+ 5324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.137108 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.137117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.137123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.137340 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.137929 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.137936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.137941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.138357 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:26.138734 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:26.140077 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0929 01:00:26.140363 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 01:00:26.141551 4488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49742
W20260929 01:00:26.142195 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:26.142393 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4357
I20260929 01:00:26.168728 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4497
Sep 29 01:00:26 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:26.184345 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.169483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.3.21.194:42671)
0929 01:00:26.170052 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:26.170060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:26.170124 (+ 64us) server_negotiation.cc:407] Connection header received
0929 01:00:26.170345 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:26.170351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:26.170434 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:26.170572 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:26.171717 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.172442 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.173671 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.174093 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.174663 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:26.178468 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:26.178532 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:26.178556 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:26.178607 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:26.181484 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.182219 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.182227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.182229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.182363 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.182771 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.182778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.182781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.183123 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:26.183425 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:26.183693 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 01:00:26.183939 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":339,"threads_started":1}
W20260929 01:00:26.185072 4628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33376
W20260929 01:00:26.186184 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:26.186625 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4497
I20260929 01:00:26.204938 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4637
Sep 29 01:00:26 dist-test-slave-k0b7 krb5kdc[4241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643625, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:26.224982 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:26.205887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.3.21.195:46227)
0929 01:00:26.206128 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:26.206133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:26.206413 (+ 280us) server_negotiation.cc:407] Connection header received
0929 01:00:26.206646 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:26.206651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:26.206746 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:26.206882 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:26.208223 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.209684 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.211534 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:26.212264 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:26.212402 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:26.217016 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:26.217037 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:26.217065 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:26.217121 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:26.221486 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.222441 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.222452 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.222456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.222658 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:26.223161 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:26.223166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:26.223169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:26.223609 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:26.224007 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:26.224370 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 01:00:26.224607 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 01:00:26.225903 4768 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47630
W20260929 01:00:26.226770 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4637: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:26.226950 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4637
I20260929 01:00:26.247067 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4254
I20260929 01:00:26.248505 4326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:26.371690 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4254
2026-09-29T01:00:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5024 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:26 dist-test-slave-k0b7 krb5kdc[4863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:26 dist-test-slave-k0b7 krb5kdc[4863](info): set up 2 sockets
Sep 29 01:00:26 dist-test-slave-k0b7 krb5kdc[4863](info): commencing operation
krb5kdc: starting...
W20260929 01:00:27.511202 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:27 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:27.740852 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36599
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:28.128019 4876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:28.128362 4876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:28.128508 4876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:28.139673 4876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:28.139846 4876 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:00:28.139950 4876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:28.140064 4876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:28.140142 4876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:28.140232 4876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:28.152972 4876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36599
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:28.154417 4876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:28.155844 4876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:28.171424 4881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:28.171397 4882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:28.171797 4884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:28.172775 4876 server_base.cc:1034] running on GCE node
I20260929 01:00:28.173684 4876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:28.175480 4876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:28.176834 4876 hybrid_clock.cc:648] HybridClock initialized: now 1790643628176752 us; error 52 us; skew 500 ppm
I20260929 01:00:28.177225 4876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:28 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:28.180910 4876 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:28.183000 4876 webserver.cc:460] Webserver started at http://127.3.21.254:36527/ using document root <none> and password file <none>
I20260929 01:00:28.183771 4876 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:28.183871 4876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:28.184136 4876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:28.187976 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "64a625778cd3491c8e005c9146d44aad"
format_stamp: "Formatted at 2026-09-29 01:00:28 on dist-test-slave-k0b7"
server_key: "e932862ed4ca4e74e76fd1ade339acde"
server_key_iv: "be35f274a3895f6ef2c8448c5720268e"
server_key_version: "encryptionkey@0"
I20260929 01:00:28.188828 4876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "64a625778cd3491c8e005c9146d44aad"
format_stamp: "Formatted at 2026-09-29 01:00:28 on dist-test-slave-k0b7"
server_key: "e932862ed4ca4e74e76fd1ade339acde"
server_key_iv: "be35f274a3895f6ef2c8448c5720268e"
server_key_version: "encryptionkey@0"
I20260929 01:00:28.195380 4876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:00:28.200349 4891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:28.202569 4876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:00:28.202812 4876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "64a625778cd3491c8e005c9146d44aad"
format_stamp: "Formatted at 2026-09-29 01:00:28 on dist-test-slave-k0b7"
server_key: "e932862ed4ca4e74e76fd1ade339acde"
server_key_iv: "be35f274a3895f6ef2c8448c5720268e"
server_key_version: "encryptionkey@0"
I20260929 01:00:28.203044 4876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:28.220520 4876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:28.226962 4876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:28.227522 4876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:28.255774 4876 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36599
I20260929 01:00:28.255795 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36599 every 8 connection(s)
I20260929 01:00:28.257591 4876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:28.262169 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:28.265033 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4876
I20260929 01:00:28.265235 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:28.265973 3159 external_mini_cluster.cc:1428] Setting key c318ac04fee0645ecd45fb87c91386f4
I20260929 01:00:28.275391 4953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Bootstrap starting.
I20260929 01:00:28.279405 4953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:28.280571 4953 log.cc:824] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:00:28 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:28.285409 4953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: No bootstrap required, opened a new log
I20260929 01:00:28.293445 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.268541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.3.21.254:36599)
0929 01:00:28.269541 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:28.269556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:28.269606 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:00:28.271693 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:28.271734 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:28.272258 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:28.272752 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:28.274708 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.276251 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.277711 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.278606 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.278814 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:28.283411 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:28.283422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:28.283445 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:28.283499 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:28.287659 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:28.288790 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:28.288803 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:28.288812 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:28.289075 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:28.289606 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:28.289612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:28.289616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:28.290337 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:28.290699 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:28.291438 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 01:00:28.291907 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":500,"threads_started":1}
I20260929 01:00:28.299530 4953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a625778cd3491c8e005c9146d44aad" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36599 } }
I20260929 01:00:28.300007 4953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:28.300123 4953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a625778cd3491c8e005c9146d44aad, State: Initialized, Role: FOLLOWER
I20260929 01:00:28.300789 4953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a625778cd3491c8e005c9146d44aad" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36599 } }
I20260929 01:00:28.301069 4953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:28.301237 4953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:28.301465 4953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:28.304293 4953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a625778cd3491c8e005c9146d44aad" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36599 } }
I20260929 01:00:28.304867 4953 leader_election.cc:302] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a625778cd3491c8e005c9146d44aad; no voters:
I20260929 01:00:28.305526 4953 leader_election.cc:288] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:28.305804 4958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:28.307039 4958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [term 1 LEADER]: Becoming Leader. State: Replica: 64a625778cd3491c8e005c9146d44aad, State: Running, Role: LEADER
I20260929 01:00:28.308471 4953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:28.308395 4958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a625778cd3491c8e005c9146d44aad" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36599 } }
I20260929 01:00:28.311818 4959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a625778cd3491c8e005c9146d44aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a625778cd3491c8e005c9146d44aad" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36599 } } }
I20260929 01:00:28.312270 4959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:28.312057 4960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a625778cd3491c8e005c9146d44aad. Latest consensus state: current_term: 1 leader_uuid: "64a625778cd3491c8e005c9146d44aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a625778cd3491c8e005c9146d44aad" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36599 } } }
I20260929 01:00:28.312479 4960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:28.314543 4965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:28.320820 4965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:28.332229 4965 catalog_manager.cc:1347] Generated new cluster ID: b9793f2431bd46d48f59804e7eb640d5
I20260929 01:00:28.332397 4965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:28.344357 4965 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:28.346596 4965 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:28.357290 4965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Generated new TSK 0
I20260929 01:00:28.358599 4965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:28.420338 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36599
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:28.808949 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:28.809360 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:28.809535 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:28.818058 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:28.818238 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:28.818378 4979 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:00:28.818463 4979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:00:28.818622 4979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:28.832976 4979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:28.835156 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:28.836867 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:28.855084 4984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:28.855053 4985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:28.855449 4987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:28.855731 4979 server_base.cc:1034] running on GCE node
I20260929 01:00:28.857213 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:28.859426 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:28.860930 4979 hybrid_clock.cc:648] HybridClock initialized: now 1790643628860831 us; error 104 us; skew 500 ppm
I20260929 01:00:28.861421 4979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:28 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:28.865759 4979 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:28.868311 4979 webserver.cc:460] Webserver started at http://127.3.21.193:35553/ using document root <none> and password file <none>
I20260929 01:00:28.869195 4979 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:28.869366 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:28.869732 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:28.873617 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "1f5655a6533d4f2283d68fe903bb371b"
format_stamp: "Formatted at 2026-09-29 01:00:28 on dist-test-slave-k0b7"
server_key: "1c4d9ee491ae0fdd10fade8ec4cd7f1b"
server_key_iv: "127ae713f4baa13bcd7abf2c1ebf0cb1"
server_key_version: "encryptionkey@0"
I20260929 01:00:28.874665 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "1f5655a6533d4f2283d68fe903bb371b"
format_stamp: "Formatted at 2026-09-29 01:00:28 on dist-test-slave-k0b7"
server_key: "1c4d9ee491ae0fdd10fade8ec4cd7f1b"
server_key_iv: "127ae713f4baa13bcd7abf2c1ebf0cb1"
server_key_version: "encryptionkey@0"
I20260929 01:00:28.881978 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:00:28.888521 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:28.890961 4979 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:00:28.891263 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "1f5655a6533d4f2283d68fe903bb371b"
format_stamp: "Formatted at 2026-09-29 01:00:28 on dist-test-slave-k0b7"
server_key: "1c4d9ee491ae0fdd10fade8ec4cd7f1b"
server_key_iv: "127ae713f4baa13bcd7abf2c1ebf0cb1"
server_key_version: "encryptionkey@0"
I20260929 01:00:28.891528 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:28.917585 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:28.923341 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:28.923702 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:28.927248 4979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:28.927418 4979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:28.927595 4979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:28.927708 4979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:00:28 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:28.964362 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.928619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:55753 (local address 127.3.21.254:36599)
0929 01:00:28.929168 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:28.929178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:28.930892 (+ 1714us) server_negotiation.cc:407] Connection header received
0929 01:00:28.933918 (+ 3026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:28.933928 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:28.934065 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:28.934255 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:28.936908 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.937932 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.941063 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.941710 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.942044 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:28.948125 (+ 6081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:28.948131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:28.948134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:28.948171 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:28.958555 (+ 10384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:28.960004 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:28.960013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:28.960018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:28.960212 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:28.960871 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:28.960879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:28.960884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:28.961342 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:28.961525 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:28.962912 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0929 01:00:28.963271 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":254,"threads_started":1}
I20260929 01:00:28.964671 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.929459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.3.21.193:55753)
0929 01:00:28.930636 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:28.930711 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:28.933492 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:28.934614 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:28.934625 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:28.935538 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:28.936653 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:28.936684 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.938152 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.938157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:28.940846 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:28.940866 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.942168 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.942179 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:28.942541 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:28.943639 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:28.943713 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:28.947800 (+ 4087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:28.958797 (+ 10997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:28.958809 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:28.958838 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:28.959776 (+ 938us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:28.960396 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:28.960402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:28.960407 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:28.960676 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:28.961562 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:28.961573 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:28.962191 (+ 618us) client_negotiation.cc:769] Sending connection context
0929 01:00:28.962775 (+ 584us) client_negotiation.cc:240] Negotiation successful
0929 01:00:28.963286 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":442,"threads_started":1}
I20260929 01:00:28.998005 4979 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:36547
I20260929 01:00:28.998113 5114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:36547 every 8 connection(s)
I20260929 01:00:28.999245 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.974522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:34181 (local address 127.3.21.254:36599)
0929 01:00:28.974774 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:28.974782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:28.980034 (+ 5252us) server_negotiation.cc:407] Connection header received
0929 01:00:28.980445 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:28.980455 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:28.980593 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:28.980791 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:28.982287 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.983409 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.985751 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.986337 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.986534 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:28.989937 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:28.989946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:28.989953 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:28.990015 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:28.994409 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:28.995636 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:28.995643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:28.995650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:28.995812 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:28.996295 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:28.996300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:28.996302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:28.996699 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:28.997020 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:28.998688 (+ 1668us) server_negotiation.cc:299] Negotiation successful
0929 01:00:28.998920 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 01:00:28.999576 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:28.999662 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:28.974420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.3.21.193:34181)
0929 01:00:28.979413 (+ 4993us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:28.979789 (+ 376us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:28.980184 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:28.980896 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:28.980903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:28.981552 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:28.982023 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:28.982037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.983609 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.983619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:28.985526 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:28.985545 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:28.986630 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:28.986770 (+ 140us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:28.987177 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:28.988139 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:28.988164 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:28.989626 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:28.994676 (+ 5050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:28.994687 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:28.994691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:28.995453 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:28.995948 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:28.995953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:28.995956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:28.996154 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:28.997114 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:28.997124 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:28.997637 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 01:00:28.997863 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 01:00:28.998343 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4733,"thread_start_us":4502,"threads_started":1}
I20260929 01:00:29.002936 5115 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36599
I20260929 01:00:29.003235 5115 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:29.003963 5115 heartbeater.cc:511] Master 127.3.21.254:36599 requested a full tablet report, sending...
I20260929 01:00:29.005193 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 4979
I20260929 01:00:29.005394 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:29.006033 3159 external_mini_cluster.cc:1428] Setting key 3667b4cebb8425f73ad0f4a4eee75531
I20260929 01:00:29.006448 4908 ts_manager.cc:194] Registered new tserver with Master: 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547)
I20260929 01:00:29.009035 4908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:55753
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:30.012678 5115 heartbeater.cc:503] Master 127.3.21.254:36599 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
W20260929 01:00:30.320744 3159 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.21.194: real 1.312s user 0.001s sys 0.010s
I20260929 01:00:30.322139 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36599
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:30.688874 5119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:30.689333 5119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:30.689633 5119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:30.698711 5119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:30.698851 5119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:30.699015 5119 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:00:30.699100 5119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:00:30.699239 5119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:30.710683 5119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:30.712280 5119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:30.713986 5119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:30.732549 5125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:30.733203 5127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:30.733587 5119 server_base.cc:1034] running on GCE node
W20260929 01:00:30.733234 5124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:30.734961 5119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:30.736523 5119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:30.737869 5119 hybrid_clock.cc:648] HybridClock initialized: now 1790643630737790 us; error 82 us; skew 500 ppm
I20260929 01:00:30.738303 5119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:30 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:30.743382 5119 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:30.745532 5119 webserver.cc:460] Webserver started at http://127.3.21.194:40971/ using document root <none> and password file <none>
I20260929 01:00:30.746384 5119 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:30.746588 5119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:30.746954 5119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:30.750190 5119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "7a7976b7a0cb4b01891c9e51b7f1739b"
format_stamp: "Formatted at 2026-09-29 01:00:30 on dist-test-slave-k0b7"
server_key: "20c9e37fb510f051e1d99bc2963920e9"
server_key_iv: "b4286a3a131987cfcec34a01521f1320"
server_key_version: "encryptionkey@0"
I20260929 01:00:30.751120 5119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "7a7976b7a0cb4b01891c9e51b7f1739b"
format_stamp: "Formatted at 2026-09-29 01:00:30 on dist-test-slave-k0b7"
server_key: "20c9e37fb510f051e1d99bc2963920e9"
server_key_iv: "b4286a3a131987cfcec34a01521f1320"
server_key_version: "encryptionkey@0"
I20260929 01:00:30.757423 5119 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:00:30.762079 5134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:30.764647 5119 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:00:30.764899 5119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "7a7976b7a0cb4b01891c9e51b7f1739b"
format_stamp: "Formatted at 2026-09-29 01:00:30 on dist-test-slave-k0b7"
server_key: "20c9e37fb510f051e1d99bc2963920e9"
server_key_iv: "b4286a3a131987cfcec34a01521f1320"
server_key_version: "encryptionkey@0"
I20260929 01:00:30.765209 5119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:30.787209 5119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:30.792174 5119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:30.792507 5119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:30.796080 5119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:30.796234 5119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:30.796370 5119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:30.796479 5119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:00:30 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:30.822394 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.796796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:55489 (local address 127.3.21.254:36599)
0929 01:00:30.799348 (+ 2552us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:30.799355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:30.799380 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:00:30.801020 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:30.801027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:30.801125 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:30.801247 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:30.803690 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.804491 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.807931 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.808673 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.808804 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:30.814436 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:30.814445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:30.814452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:30.814527 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:30.817689 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:30.818676 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:30.818679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:30.818682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:30.818830 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:30.819470 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:30.819473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:30.819475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:30.819838 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:30.819987 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:30.821320 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0929 01:00:30.821684 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2414,"thread_start_us":2303,"threads_started":1}
I20260929 01:00:30.822901 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.797513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.3.21.194:55489)
0929 01:00:30.798449 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:30.798542 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:30.800644 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:30.801564 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:30.801578 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:30.802535 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:30.803477 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:30.803499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.804705 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.804712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:30.807694 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:30.807717 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.808936 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.808944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:30.809261 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:30.810729 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:30.810769 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:30.814117 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:30.817871 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:30.817881 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:30.817903 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:30.818525 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:30.818960 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:30.818965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:30.818972 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:30.819304 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:30.820070 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:30.820078 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:30.820698 (+ 620us) client_negotiation.cc:769] Sending connection context
0929 01:00:30.821098 (+ 400us) client_negotiation.cc:240] Negotiation successful
0929 01:00:30.821582 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":372,"threads_started":1}
I20260929 01:00:30.848560 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.830829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.3.21.194:42407)
0929 01:00:30.831803 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:30.831821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:30.832022 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:30.832386 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:30.832390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:30.832955 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:30.833277 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:30.833315 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.834370 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.834376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:30.835844 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:30.835861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.837304 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.837317 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:30.837485 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:30.838251 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:30.838272 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:30.839878 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:30.844882 (+ 5004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:30.844895 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:30.844898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:30.845624 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:30.846083 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:30.846088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:30.846092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:30.846248 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:30.847188 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:30.847198 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:30.847425 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 01:00:30.847765 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 01:00:30.848181 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":566,"threads_started":1}
I20260929 01:00:30.849493 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:30.830938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:42407 (local address 127.3.21.254:36599)
0929 01:00:30.831212 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:30.831219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:30.831926 (+ 707us) server_negotiation.cc:407] Connection header received
0929 01:00:30.832162 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:30.832168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:30.832244 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:30.832356 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:30.833469 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.834211 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.836133 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:30.837070 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:30.837274 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:30.840182 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:30.840189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:30.840196 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:30.840266 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:30.844673 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:30.845781 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:30.845787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:30.845791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:30.845958 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:30.846372 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:30.846397 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:30.846399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:30.846818 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:30.847234 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:30.848952 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0929 01:00:30.849160 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 01:00:30.867213 5119 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:43797
I20260929 01:00:30.867257 5254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:43797 every 8 connection(s)
I20260929 01:00:30.868644 5119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:30.869699 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 5119
I20260929 01:00:30.869918 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:30.870467 3159 external_mini_cluster.cc:1428] Setting key 0ae3c9559f3ada7bcbf3b1e8bc130ac3
I20260929 01:00:30.871417 5255 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36599
I20260929 01:00:30.871701 5255 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:30.872583 5255 heartbeater.cc:511] Master 127.3.21.254:36599 requested a full tablet report, sending...
I20260929 01:00:30.874383 4908 ts_manager.cc:194] Registered new tserver with Master: 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797)
I20260929 01:00:30.875859 4908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:55489
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:00:31.879560 5255 heartbeater.cc:503] Master 127.3.21.254:36599 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
W20260929 01:00:32.166393 3159 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.21.195: real 1.293s user 0.000s sys 0.010s
I20260929 01:00:32.172039 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36599
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:32.561733 5259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:32.562067 5259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:32.562252 5259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:32.570269 5259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:32.570407 5259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:32.570585 5259 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:00:32.570667 5259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:00:32.570797 5259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:32.581719 5259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:32.583246 5259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:32.584703 5259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:32.600011 5264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:32.599949 5265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:32.599943 5267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:32.600610 5259 server_base.cc:1034] running on GCE node
I20260929 01:00:32.601392 5259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:32.602459 5259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:32.603763 5259 hybrid_clock.cc:648] HybridClock initialized: now 1790643632603692 us; error 69 us; skew 500 ppm
I20260929 01:00:32.604178 5259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:32 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:32.608932 5259 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:32.610860 5259 webserver.cc:460] Webserver started at http://127.3.21.195:36615/ using document root <none> and password file <none>
I20260929 01:00:32.611657 5259 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:32.611811 5259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:32.612149 5259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:32.615847 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "5294096528d04a1b858c00d34f0f089b"
format_stamp: "Formatted at 2026-09-29 01:00:32 on dist-test-slave-k0b7"
server_key: "308dee4bed3e6602b2ee2d3b5d00313a"
server_key_iv: "b7b6d6d9d5b549b0c2f004c41d033130"
server_key_version: "encryptionkey@0"
I20260929 01:00:32.616706 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "5294096528d04a1b858c00d34f0f089b"
format_stamp: "Formatted at 2026-09-29 01:00:32 on dist-test-slave-k0b7"
server_key: "308dee4bed3e6602b2ee2d3b5d00313a"
server_key_iv: "b7b6d6d9d5b549b0c2f004c41d033130"
server_key_version: "encryptionkey@0"
I20260929 01:00:32.623168 5259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:00:32.627749 5274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:32.629621 5259 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:32.629792 5259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "5294096528d04a1b858c00d34f0f089b"
format_stamp: "Formatted at 2026-09-29 01:00:32 on dist-test-slave-k0b7"
server_key: "308dee4bed3e6602b2ee2d3b5d00313a"
server_key_iv: "b7b6d6d9d5b549b0c2f004c41d033130"
server_key_version: "encryptionkey@0"
I20260929 01:00:32.630005 5259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:32.651476 5259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:32.656704 5259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:32.656972 5259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:32.659806 5259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:32.659960 5259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:32.660110 5259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:32.660219 5259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:00:32 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:32.692435 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.661009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:40807 (local address 127.3.21.254:36599)
0929 01:00:32.661498 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:32.661503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:32.663330 (+ 1827us) server_negotiation.cc:407] Connection header received
0929 01:00:32.666624 (+ 3294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:32.666634 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:32.666757 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:32.666855 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:32.670193 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.671466 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.675101 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.675567 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.675727 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.683262 (+ 7535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:32.683269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:32.683272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:32.683312 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:32.687665 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:32.688956 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:32.688965 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:32.688974 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:32.689193 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:32.690033 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:32.690041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:32.690047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:32.690571 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:32.690737 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:32.691643 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 01:00:32.691903 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
I20260929 01:00:32.693598 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.661675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.3.21.195:40807)
0929 01:00:32.663054 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.663097 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:32.666062 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:32.667566 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:32.667576 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:32.668669 (+ 1093us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:32.669875 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.669910 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.671726 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.671737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.674873 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.674896 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.676374 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.676389 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.676885 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.678159 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:32.678203 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:32.682931 (+ 4728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:32.687903 (+ 4972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:32.687917 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:32.687950 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:32.688778 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:32.689399 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:32.689410 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:32.689417 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:32.689842 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:32.690732 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:32.690743 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:32.691221 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 01:00:32.691529 (+ 308us) client_negotiation.cc:240] Negotiation successful
0929 01:00:32.692166 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":410,"threads_started":1}
I20260929 01:00:32.722818 5259 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:36331
I20260929 01:00:32.722843 5394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:36331 every 8 connection(s)
I20260929 01:00:32.724273 5259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:32.724642 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.702653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:48485 (local address 127.3.21.254:36599)
0929 01:00:32.702934 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:32.702944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:32.707721 (+ 4777us) server_negotiation.cc:407] Connection header received
0929 01:00:32.707967 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:32.707974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:32.708078 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:32.708238 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:32.710410 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.711796 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.714758 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.715395 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.715614 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.717809 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:32.717814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:32.717817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:32.717850 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:32.721123 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:32.722144 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:32.722151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:32.722155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:32.722327 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:32.722839 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:32.722846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:32.722850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:32.723303 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:32.723511 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:32.723971 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 01:00:32.724217 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:00:32.724798 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.703199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.3.21.195:48485)
0929 01:00:32.707539 (+ 4340us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.707568 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:32.707758 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:32.708280 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:32.708288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:32.709478 (+ 1190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:32.710080 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.710119 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.712060 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.712066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.714369 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.714393 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.715684 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.715716 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.715888 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.716797 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:32.716816 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:32.717577 (+ 761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:32.721335 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:32.721341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:32.721346 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:32.721948 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:32.722505 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:32.722513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:32.722522 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:32.722698 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:32.723446 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:32.723451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:32.723686 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 01:00:32.723856 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 01:00:32.724240 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4156,"thread_start_us":4014,"threads_started":1}
I20260929 01:00:32.725764 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 5259
I20260929 01:00:32.726003 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:32.726759 3159 external_mini_cluster.cc:1428] Setting key 1aa7c461c7144c2898c40711772a1b10
I20260929 01:00:32.729208 5395 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36599
I20260929 01:00:32.730370 5395 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:32.731096 5395 heartbeater.cc:511] Master 127.3.21.254:36599 requested a full tablet report, sending...
I20260929 01:00:32.732777 4907 ts_manager.cc:194] Registered new tserver with Master: 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:32.733848 4907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:40807
I20260929 01:00:32.745095 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:00:32 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:00:32 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:32.841816 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.824737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.3.21.254:36599)
0929 01:00:32.824944 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:32.824956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:32.825309 (+ 353us) server_negotiation.cc:407] Connection header received
0929 01:00:32.825464 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:32.825470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:32.825587 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:32.825736 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:32.827522 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.828489 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.829997 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.830473 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.830644 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.835171 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:32.835176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:32.835180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:32.835224 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:32.838320 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:32.839282 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:32.839286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:32.839289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:32.839455 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:32.840090 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:32.840094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:32.840097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:32.840406 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:32.840716 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:32.841245 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 01:00:32.841524 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:00:32.847626 4907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:32.855635 4907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:32.893663 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.870700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.3.21.195:36331)
0929 01:00:32.878929 (+ 8229us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:32.878943 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:32.882586 (+ 3643us) server_negotiation.cc:407] Connection header received
0929 01:00:32.882740 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:32.882753 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:32.883603 (+ 850us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:32.883860 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:32.885101 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.886976 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.890882 (+ 3906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.892152 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.892287 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.892573 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:32.893142 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 01:00:32.893315 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7963,"thread_start_us":7718,"threads_started":1}
I20260929 01:00:32.893741 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.869922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:36331 (local address 127.0.0.1:48998)
0929 01:00:32.881808 (+ 11886us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.881833 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:32.882192 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:32.884228 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:32.884237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:32.884275 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:32.884823 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.884840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.887180 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.887186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.890695 (+ 3509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.890706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.892372 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.892379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.892647 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.892734 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 01:00:32.892892 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 01:00:32.893204 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11683,"mutex_wait_us":882,"thread_start_us":10549,"threads_started":1}
I20260929 01:00:32.893899 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.869655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36547 (local address 127.0.0.1:49228)
0929 01:00:32.880916 (+ 11261us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.880960 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:32.881209 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:32.884228 (+ 3019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:32.884237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:32.884287 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:32.884940 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.884954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.887180 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.887186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.890164 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.890179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.892884 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.892890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.893101 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.893144 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:00:32.893231 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 01:00:32.893385 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10964,"mutex_wait_us":313,"spinlock_wait_cycles":2816,"thread_start_us":18497,"threads_started":1}
I20260929 01:00:32.895614 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.869981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.3.21.193:36547)
0929 01:00:32.883383 (+ 13402us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:32.883394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:32.883423 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:00:32.883545 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:32.883553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:32.883756 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:32.883936 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:32.885115 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.886976 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.890367 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.892086 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.894321 (+ 2235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.894644 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:32.894739 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 01:00:32.894922 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13212,"thread_start_us":14304,"threads_started":1}
I20260929 01:00:32.897949 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.871167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.3.21.194:43797)
0929 01:00:32.886787 (+ 15620us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:32.886801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:32.889268 (+ 2467us) server_negotiation.cc:407] Connection header received
0929 01:00:32.889393 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:32.889403 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:32.889763 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:32.890014 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:32.890871 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.892373 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.895053 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.896651 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.896800 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.897011 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:32.897502 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 01:00:32.897663 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15355,"thread_start_us":15723,"threads_started":1}
I20260929 01:00:32.897986 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:32.870555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43797 (local address 127.0.0.1:33970)
0929 01:00:32.888494 (+ 17939us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:32.888515 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:32.888672 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:32.890000 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:32.890005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:32.890037 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:32.890657 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.890672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.892558 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.892563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.894864 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:32.894879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:32.897157 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:32.897165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:32.897305 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:32.897329 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:00:32.897447 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 01:00:32.897593 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17786,"mutex_wait_us":3,"thread_start_us":10871,"threads_started":1}
I20260929 01:00:32.898955 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 (DEFAULT_TABLE table=test-table [id=578bf3b5ce0f4bdbb8089347917ab960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:32.899658 5045 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 (DEFAULT_TABLE table=test-table [id=578bf3b5ce0f4bdbb8089347917ab960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:32.900593 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:32.901106 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:32.901645 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 (DEFAULT_TABLE table=test-table [id=578bf3b5ce0f4bdbb8089347917ab960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:32.902846 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:32.915565 5414 tablet_bootstrap.cc:492] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Bootstrap starting.
I20260929 01:00:32.919930 5415 tablet_bootstrap.cc:492] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Bootstrap starting.
I20260929 01:00:32.920557 5414 tablet_bootstrap.cc:654] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:32.922014 5414 log.cc:824] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:32.923290 5416 tablet_bootstrap.cc:492] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Bootstrap starting.
I20260929 01:00:32.926451 5415 tablet_bootstrap.cc:654] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:32.927206 5416 tablet_bootstrap.cc:654] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:32.927416 5414 tablet_bootstrap.cc:492] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: No bootstrap required, opened a new log
I20260929 01:00:32.927698 5414 ts_tablet_manager.cc:1397] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 01:00:32.927919 5415 log.cc:824] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:32.928426 5416 log.cc:824] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:32.931731 5415 tablet_bootstrap.cc:492] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: No bootstrap required, opened a new log
I20260929 01:00:32.932039 5415 ts_tablet_manager.cc:1397] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260929 01:00:32.932294 5416 tablet_bootstrap.cc:492] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: No bootstrap required, opened a new log
I20260929 01:00:32.932564 5416 ts_tablet_manager.cc:1397] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 01:00:32.940024 5414 raft_consensus.cc:348] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:32.939970 5415 raft_consensus.cc:348] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:32.940518 5414 raft_consensus.cc:374] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:32.940522 5415 raft_consensus.cc:374] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:32.940622 5414 raft_consensus.cc:729] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Initialized, Role: FOLLOWER
I20260929 01:00:32.940632 5415 raft_consensus.cc:729] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5294096528d04a1b858c00d34f0f089b, State: Initialized, Role: FOLLOWER
I20260929 01:00:32.941283 5415 consensus_queue.cc:260] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:32.941385 5414 consensus_queue.cc:260] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:32.942536 5395 heartbeater.cc:503] Master 127.3.21.254:36599 was elected leader, sending a full tablet report...
I20260929 01:00:32.943295 5415 ts_tablet_manager.cc:1428] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 01:00:32.944036 5416 raft_consensus.cc:348] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:32.944525 5416 raft_consensus.cc:374] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:32.944615 5416 raft_consensus.cc:729] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Initialized, Role: FOLLOWER
I20260929 01:00:32.945366 5414 ts_tablet_manager.cc:1428] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent starting tablet: real 0.018s user 0.006s sys 0.011s
I20260929 01:00:32.945389 5416 consensus_queue.cc:260] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:32.948107 5416 ts_tablet_manager.cc:1428] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
W20260929 01:00:32.979286 5396 tablet.cc:2369] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:33.005164 5116 tablet.cc:2369] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:33.122459 5256 tablet.cc:2369] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:33.192517 5421 raft_consensus.cc:482] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:33.192936 5421 raft_consensus.cc:504] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:33.194904 5421 leader_election.cc:288] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:33.202952 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.195363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:50235 (local address 127.3.21.193:36547)
0929 01:00:33.195566 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.195572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.196646 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 01:00:33.196891 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.196897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.196988 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.197114 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:33.198210 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.199161 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.201177 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.201976 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.202182 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.202262 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.202550 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.202677 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:00:33.203146 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.195443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36547 (local address 127.3.21.194:50235)
0929 01:00:33.196423 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.196444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.196650 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.197164 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.197169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.197274 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:33.197957 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.197968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.199298 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.199304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.200998 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.201015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.202197 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.202203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.202367 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.202394 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.202526 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.202675 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":44,"spinlock_wait_cycles":57216,"thread_start_us":650,"threads_started":1}
I20260929 01:00:33.203267 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.195377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:45309 (local address 127.3.21.195:36331)
0929 01:00:33.195566 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.195572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.196889 (+ 1317us) server_negotiation.cc:407] Connection header received
0929 01:00:33.196974 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.196979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.197072 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.197169 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:33.198117 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.199030 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.201282 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.202135 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.202350 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.202437 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.202797 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.202931 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:00:33.203485 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.195443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:36331 (local address 127.3.21.194:45309)
0929 01:00:33.196681 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.196698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.196819 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.197167 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.197170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.197219 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:33.197905 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.197921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.199187 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.199193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.201092 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.201105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.202390 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.202397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.202612 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.202637 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.202751 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.202899 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":214,"thread_start_us":464,"threads_started":1}
I20260929 01:00:33.204422 5065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8e162a6a3a418dbdf5af52bc8db7f6" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5655a6533d4f2283d68fe903bb371b" is_pre_election: true
I20260929 01:00:33.204757 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8e162a6a3a418dbdf5af52bc8db7f6" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b" is_pre_election: true
I20260929 01:00:33.205106 5065 raft_consensus.cc:2454] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 0.
I20260929 01:00:33.205229 5345 raft_consensus.cc:2454] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 0.
I20260929 01:00:33.205940 5138 leader_election.cc:302] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 5294096528d04a1b858c00d34f0f089b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:33.206418 5421 raft_consensus.cc:2781] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:33.206601 5421 raft_consensus.cc:482] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:33.206737 5421 raft_consensus.cc:3037] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.209108 5421 raft_consensus.cc:504] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:33.209791 5421 leader_election.cc:288] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 election: Requested vote from peers 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:33.210340 5065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8e162a6a3a418dbdf5af52bc8db7f6" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5655a6533d4f2283d68fe903bb371b"
I20260929 01:00:33.210541 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8e162a6a3a418dbdf5af52bc8db7f6" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b"
I20260929 01:00:33.210758 5065 raft_consensus.cc:3037] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.210757 5345 raft_consensus.cc:3037] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.214147 5345 raft_consensus.cc:2454] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 1.
I20260929 01:00:33.214147 5065 raft_consensus.cc:2454] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 1.
I20260929 01:00:33.214720 5138 leader_election.cc:302] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:33.215139 5421 raft_consensus.cc:2781] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:33.215633 5421 raft_consensus.cc:686] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 LEADER]: Becoming Leader. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Running, Role: LEADER
I20260929 01:00:33.216298 5421 consensus_queue.cc:237] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } }
I20260929 01:00:33.221957 4906 catalog_manager.cc:5495] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194). New cstate: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } health_report { overall_health: HEALTHY } } }
Sep 29 01:00:33 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:33.317444 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.301247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36599 (local address 127.0.0.1:52148)
0929 01:00:33.301940 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.301957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.302100 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.302621 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.302625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.303476 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:33.303743 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.303752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.304905 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.304909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.305879 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.305891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.306980 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.306986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.307159 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.307967 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:33.307991 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:33.310622 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:33.314420 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:33.314435 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:33.314442 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:33.315171 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:33.315599 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:33.315603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:33.315605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:33.315807 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:33.316414 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:33.316424 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:33.316618 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.316762 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.316991 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":402,"threads_started":1}
I20260929 01:00:33.317533 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.301235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.3.21.254:36599)
0929 01:00:33.301451 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.301462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.302061 (+ 599us) server_negotiation.cc:407] Connection header received
0929 01:00:33.302209 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.302213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.302273 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.302363 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:33.303903 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.304720 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.306083 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.306726 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.306857 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.310832 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:33.310837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:33.310841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:33.310895 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:33.314237 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:33.315303 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:33.315307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:33.315309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:33.315436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:33.315944 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:33.315947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:33.315949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:33.316245 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:33.316369 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.316911 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.317110 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":76}
I20260929 01:00:33.323661 4906 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:52148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:00:33.324823 4906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:33.332401 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5525a1996348adb2fef063b970cc61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cb6bcd584e049719c5313959e707a5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:00:33.332629 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5525a1996348adb2fef063b970cc61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cb6bcd584e049719c5313959e707a5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:00:33.332346 5045 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5525a1996348adb2fef063b970cc61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cb6bcd584e049719c5313959e707a5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:00:33.333415 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5525a1996348adb2fef063b970cc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:33.333416 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5525a1996348adb2fef063b970cc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:33.333428 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5525a1996348adb2fef063b970cc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:33.339553 5414 tablet_bootstrap.cc:492] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: Bootstrap starting.
I20260929 01:00:33.340713 5415 tablet_bootstrap.cc:492] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: Bootstrap starting.
I20260929 01:00:33.341064 5414 tablet_bootstrap.cc:654] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:33.341259 5416 tablet_bootstrap.cc:492] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: Bootstrap starting.
I20260929 01:00:33.342599 5415 tablet_bootstrap.cc:654] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:33.343015 5416 tablet_bootstrap.cc:654] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:33.345068 5415 tablet_bootstrap.cc:492] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: No bootstrap required, opened a new log
I20260929 01:00:33.345237 5415 ts_tablet_manager.cc:1397] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260929 01:00:33.345930 5415 raft_consensus.cc:348] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.346179 5415 raft_consensus.cc:374] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:33.346302 5415 raft_consensus.cc:729] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5294096528d04a1b858c00d34f0f089b, State: Initialized, Role: FOLLOWER
I20260929 01:00:33.346599 5415 consensus_queue.cc:260] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.347208 5415 ts_tablet_manager.cc:1428] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 01:00:33.347067 5420 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.347400 5420 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.348646 5414 tablet_bootstrap.cc:492] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: No bootstrap required, opened a new log
I20260929 01:00:33.348877 5414 ts_tablet_manager.cc:1397] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260929 01:00:33.350054 5414 raft_consensus.cc:348] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.350334 5414 raft_consensus.cc:374] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:33.350443 5414 raft_consensus.cc:729] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Initialized, Role: FOLLOWER
I20260929 01:00:33.350744 5414 consensus_queue.cc:260] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.351187 5421 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.351388 5414 ts_tablet_manager.cc:1428] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:33.351511 5421 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.355108 5416 tablet_bootstrap.cc:492] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: No bootstrap required, opened a new log
I20260929 01:00:33.355257 5416 ts_tablet_manager.cc:1397] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260929 01:00:33.355815 5416 raft_consensus.cc:348] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.356009 5416 raft_consensus.cc:374] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:33.356071 5416 raft_consensus.cc:729] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Initialized, Role: FOLLOWER
I20260929 01:00:33.356302 5416 consensus_queue.cc:260] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.356822 5416 ts_tablet_manager.cc:1428] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:00:33.356971 5422 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.357431 5422 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.433357 5421 raft_consensus.cc:482] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:33.433595 5421 raft_consensus.cc:504] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.434284 5421 leader_election.cc:288] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:33.434959 5065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5525a1996348adb2fef063b970cc61" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5655a6533d4f2283d68fe903bb371b" is_pre_election: true
I20260929 01:00:33.435042 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5525a1996348adb2fef063b970cc61" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b" is_pre_election: true
I20260929 01:00:33.435297 5065 raft_consensus.cc:2454] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 0.
I20260929 01:00:33.435326 5345 raft_consensus.cc:2454] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 0.
I20260929 01:00:33.435770 5138 leader_election.cc:302] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:33.436156 5421 raft_consensus.cc:2781] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:33.436251 5421 raft_consensus.cc:482] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:33.436372 5421 raft_consensus.cc:3037] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.438970 5421 raft_consensus.cc:504] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.439745 5421 leader_election.cc:288] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 election: Requested vote from peers 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:33.440315 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5525a1996348adb2fef063b970cc61" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b"
I20260929 01:00:33.440346 5065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5525a1996348adb2fef063b970cc61" candidate_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5655a6533d4f2283d68fe903bb371b"
I20260929 01:00:33.440555 5345 raft_consensus.cc:3037] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.440610 5065 raft_consensus.cc:3037] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.443025 5065 raft_consensus.cc:2454] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 1.
I20260929 01:00:33.443567 5345 raft_consensus.cc:2454] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a7976b7a0cb4b01891c9e51b7f1739b in term 1.
I20260929 01:00:33.443599 5138 leader_election.cc:302] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:33.444073 5421 raft_consensus.cc:2781] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:33.444293 5421 raft_consensus.cc:686] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 LEADER]: Becoming Leader. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Running, Role: LEADER
I20260929 01:00:33.444754 5421 consensus_queue.cc:237] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.446375 5425 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: TxnStatusTablet state changed. Reason: New leader 7a7976b7a0cb4b01891c9e51b7f1739b. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.446691 5425 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:00:33.447379 5437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:00:33.447731 5437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:00:33.448274 4907 catalog_manager.cc:5495] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194). New cstate: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:33.479238 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.472307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43797 (local address 127.0.0.1:33982)
0929 01:00:33.472567 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.472590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.472756 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.473897 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.473901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.473915 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:33.474152 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.474162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.475427 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.475435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.476472 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.476485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.477267 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.477273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.477464 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.477545 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:33.478621 (+ 1076us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:33.478643 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.478758 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.478931 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 01:00:33.479310 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.472537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.3.21.194:43797)
0929 01:00:33.473504 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.473508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.473528 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:00:33.473608 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.473611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.473701 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.473875 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:33.474315 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.475283 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.476633 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.477092 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.477242 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.477790 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:33.478445 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:33.478649 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.478808 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.478896 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":699,"threads_started":1}
I20260929 01:00:33.487347 5065 raft_consensus.cc:1261] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Refusing update from remote peer 7a7976b7a0cb4b01891c9e51b7f1739b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:33.487342 5345 raft_consensus.cc:1261] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Refusing update from remote peer 7a7976b7a0cb4b01891c9e51b7f1739b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:33.488782 5425 consensus_queue.cc:1035] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:33.489292 5421 consensus_queue.cc:1035] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:33.498170 5422 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: TxnStatusTablet state changed. Reason: New leader 7a7976b7a0cb4b01891c9e51b7f1739b. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.498499 5422 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.503473 5441 mvcc.cc:205] Tried to move back new op lower bound from 7334476322753290240 to 7334476322591985664. Current Snapshot: MvccSnapshot[applied={T|T < 7334476322753290240}]
I20260929 01:00:33.503870 5425 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.504762 5422 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.504995 5422 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 1f5655a6533d4f2283d68fe903bb371b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.506217 5425 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:00:33.511895 5420 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: TxnStatusTablet state changed. Reason: New leader 7a7976b7a0cb4b01891c9e51b7f1739b. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.512219 5420 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.517776 5421 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.518126 5421 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 7a7976b7a0cb4b01891c9e51b7f1739b: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:00:33.529399 5420 tablet_replica.cc:439] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } } }
I20260929 01:00:33.529709 5420 tablet_replica.cc:442] T 6c5525a1996348adb2fef063b970cc61 P 5294096528d04a1b858c00d34f0f089b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:00:33.538024 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.518271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.3.21.194:43797)
0929 01:00:33.518436 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.518442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.531088 (+ 12646us) server_negotiation.cc:407] Connection header received
0929 01:00:33.531408 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.531414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.531518 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.531608 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:33.532707 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.534076 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.535707 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.536486 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.536624 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.537077 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:33.537211 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:33.537318 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.537614 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.537697 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 01:00:33.558712 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.551010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:58341 (local address 127.3.21.194:43797)
0929 01:00:33.551258 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.551266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.551984 (+ 718us) server_negotiation.cc:407] Connection header received
0929 01:00:33.552195 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.552199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.552287 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.552368 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:33.553285 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.554102 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.556586 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.557046 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.557161 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.557830 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:33.557943 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:33.558023 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.558193 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.558259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:00:33.559043 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.550994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43797 (local address 127.3.21.194:58341)
0929 01:00:33.551840 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.551862 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.552059 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.552427 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.552433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.552451 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:33.553035 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.553055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.554245 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.554252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.556364 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.556383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.557259 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.557271 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.557478 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.557569 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:33.558059 (+ 490us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:33.558071 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.558167 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.558337 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":447,"threads_started":1}
I20260929 01:00:33.570010 5065 raft_consensus.cc:1261] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Refusing update from remote peer 7a7976b7a0cb4b01891c9e51b7f1739b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:33.570001 5345 raft_consensus.cc:1261] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Refusing update from remote peer 7a7976b7a0cb4b01891c9e51b7f1739b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:33.570672 5425 consensus_queue.cc:1035] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:33.571244 5449 consensus_queue.cc:1035] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:33.653569 4907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:33.653900 4907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:33.657763 4907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Sending DeleteTablet for 3 replicas of tablet 5b8e162a6a3a418dbdf5af52bc8db7f6
I20260929 01:00:33.659005 5045 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:33 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49228
I20260929 01:00:33.659262 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:33 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48998
I20260929 01:00:33.659396 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:33 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33970
I20260929 01:00:33.660619 4907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:33.661470 4907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:33.672741 5190 tablet_service.cc:1467] Processing CreateTablet for tablet d722342ce80e401c8b7cbb9a2fc06d1a (DEFAULT_TABLE table=test-table [id=2b1bae02308d431e94810ecf3969ce5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:33.672732 5324 tablet_service.cc:1467] Processing CreateTablet for tablet d722342ce80e401c8b7cbb9a2fc06d1a (DEFAULT_TABLE table=test-table [id=2b1bae02308d431e94810ecf3969ce5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:33.673269 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d722342ce80e401c8b7cbb9a2fc06d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:33.673295 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d722342ce80e401c8b7cbb9a2fc06d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:33.679649 5044 tablet_service.cc:1467] Processing CreateTablet for tablet d722342ce80e401c8b7cbb9a2fc06d1a (DEFAULT_TABLE table=test-table [id=2b1bae02308d431e94810ecf3969ce5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:33.680334 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d722342ce80e401c8b7cbb9a2fc06d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:33.683038 5414 tablet_bootstrap.cc:492] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Bootstrap starting.
I20260929 01:00:33.684960 5414 tablet_bootstrap.cc:654] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:33.689807 5416 tablet_bootstrap.cc:492] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Bootstrap starting.
I20260929 01:00:33.690377 5415 tablet_bootstrap.cc:492] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Bootstrap starting.
I20260929 01:00:33.692291 5415 tablet_bootstrap.cc:654] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:33.692287 5416 tablet_bootstrap.cc:654] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:33.698153 5459 tablet_replica.cc:330] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: stopping tablet replica
I20260929 01:00:33.699333 5415 tablet_bootstrap.cc:492] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: No bootstrap required, opened a new log
I20260929 01:00:33.699374 5460 tablet_replica.cc:330] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: stopping tablet replica
I20260929 01:00:33.699684 5415 ts_tablet_manager.cc:1397] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 01:00:33.699952 5460 raft_consensus.cc:2229] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:33.700368 5460 raft_consensus.cc:2258] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:33.700781 5415 raft_consensus.cc:348] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.701160 5415 raft_consensus.cc:374] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:33.701296 5415 raft_consensus.cc:729] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5294096528d04a1b858c00d34f0f089b, State: Initialized, Role: FOLLOWER
I20260929 01:00:33.701645 5415 consensus_queue.cc:260] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.703260 5415 ts_tablet_manager.cc:1428] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 01:00:33.703439 5416 tablet_bootstrap.cc:492] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: No bootstrap required, opened a new log
I20260929 01:00:33.704452 5416 ts_tablet_manager.cc:1397] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260929 01:00:33.705299 5416 raft_consensus.cc:348] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.705634 5416 raft_consensus.cc:374] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:33.705776 5416 raft_consensus.cc:729] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Initialized, Role: FOLLOWER
I20260929 01:00:33.706121 5416 consensus_queue.cc:260] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.706882 5416 ts_tablet_manager.cc:1428] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:33.707456 5459 raft_consensus.cc:2229] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:33.707957 5459 raft_consensus.cc:2258] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:33.708621 5414 tablet_bootstrap.cc:492] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: No bootstrap required, opened a new log
I20260929 01:00:33.708813 5414 ts_tablet_manager.cc:1397] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20260929 01:00:33.709627 5414 raft_consensus.cc:348] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.709888 5414 raft_consensus.cc:374] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:33.709976 5414 raft_consensus.cc:729] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Initialized, Role: FOLLOWER
I20260929 01:00:33.710218 5414 consensus_queue.cc:260] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.709790 5460 ts_tablet_manager.cc:1905] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:33.712286 5414 ts_tablet_manager.cc:1428] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:00:33.718775 5461 tablet_replica.cc:330] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: stopping tablet replica
I20260929 01:00:33.719146 5460 ts_tablet_manager.cc:1918] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:33.719333 5461 raft_consensus.cc:2229] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:33.719411 5460 log.cc:1196] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/5b8e162a6a3a418dbdf5af52bc8db7f6
I20260929 01:00:33.720005 5460 ts_tablet_manager.cc:1939] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 5294096528d04a1b858c00d34f0f089b: Deleting consensus metadata
I20260929 01:00:33.720003 5461 raft_consensus.cc:2258] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:33.720916 5459 ts_tablet_manager.cc:1905] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:33.721923 4892 catalog_manager.cc:4841] TS 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331): tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 (table test-table [id=578bf3b5ce0f4bdbb8089347917ab960]) successfully deleted
I20260929 01:00:33.724839 5461 ts_tablet_manager.cc:1905] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:33.729245 5459 ts_tablet_manager.cc:1918] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:33.729583 5459 log.cc:1196] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/5b8e162a6a3a418dbdf5af52bc8db7f6
I20260929 01:00:33.730192 5459 ts_tablet_manager.cc:1939] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 1f5655a6533d4f2283d68fe903bb371b: Deleting consensus metadata
I20260929 01:00:33.732824 4895 catalog_manager.cc:4841] TS 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547): tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 (table test-table [id=578bf3b5ce0f4bdbb8089347917ab960]) successfully deleted
I20260929 01:00:33.734815 5461 ts_tablet_manager.cc:1918] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:33.735023 5461 log.cc:1196] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/5b8e162a6a3a418dbdf5af52bc8db7f6
I20260929 01:00:33.735517 5461 ts_tablet_manager.cc:1939] T 5b8e162a6a3a418dbdf5af52bc8db7f6 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting consensus metadata
I20260929 01:00:33.737083 4894 catalog_manager.cc:4841] TS 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797): tablet 5b8e162a6a3a418dbdf5af52bc8db7f6 (table test-table [id=578bf3b5ce0f4bdbb8089347917ab960]) successfully deleted
I20260929 01:00:33.896092 5422 raft_consensus.cc:482] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:33.896353 5422 raft_consensus.cc:504] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.898268 5422 leader_election.cc:288] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:33.917145 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.899266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:46219 (local address 127.3.21.194:43797)
0929 01:00:33.899539 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.899548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.910419 (+ 10871us) server_negotiation.cc:407] Connection header received
0929 01:00:33.910729 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.910734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.910808 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.910932 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:33.911507 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.912502 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.914891 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.916079 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.916244 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.916330 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.916697 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.916845 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 01:00:33.917235 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.898532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43797 (local address 127.3.21.193:46219)
0929 01:00:33.910225 (+ 11693us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.910260 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.910579 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.910968 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.910972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.910997 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:33.911282 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.911296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.912701 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.912706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.914692 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.914723 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.916270 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.916277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.916486 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.916523 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.916635 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.916803 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11500,"mutex_wait_us":1,"thread_start_us":11839,"threads_started":1}
I20260929 01:00:33.918383 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d722342ce80e401c8b7cbb9a2fc06d1a" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" is_pre_election: true
I20260929 01:00:33.918727 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.898732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:36331 (local address 127.3.21.193:51295)
0929 01:00:33.910734 (+ 12002us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:33.910758 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:33.910930 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:33.911426 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:33.911432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:33.911470 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:33.911862 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.911876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.913328 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.913333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.915759 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:33.915775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.917819 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.917828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:33.918036 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.918104 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 01:00:33.918204 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 01:00:33.918363 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11858,"thread_start_us":11167,"threads_started":1}
I20260929 01:00:33.918812 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:33.898858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:51295 (local address 127.3.21.195:36331)
0929 01:00:33.909625 (+ 10767us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:33.909635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:33.910876 (+ 1241us) server_negotiation.cc:407] Connection header received
0929 01:00:33.911170 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:33.911177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:33.911267 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:33.911377 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:33.912090 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.913166 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.915952 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:33.917219 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:33.917795 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:33.917874 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:33.918234 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:00:33.918402 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10526,"thread_start_us":10788,"threads_started":1}
I20260929 01:00:33.918964 5210 raft_consensus.cc:2454] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 0.
I20260929 01:00:33.919704 4997 leader_election.cc:302] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:33.919926 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d722342ce80e401c8b7cbb9a2fc06d1a" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b" is_pre_election: true
I20260929 01:00:33.920179 5345 raft_consensus.cc:2454] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 0.
I20260929 01:00:33.920308 5422 raft_consensus.cc:2781] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:33.920434 5422 raft_consensus.cc:482] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:33.920555 5422 raft_consensus.cc:3037] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.922704 5422 raft_consensus.cc:504] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.923369 5422 leader_election.cc:288] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 election: Requested vote from peers 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:33.923924 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d722342ce80e401c8b7cbb9a2fc06d1a" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b"
I20260929 01:00:33.923885 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d722342ce80e401c8b7cbb9a2fc06d1a" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b"
I20260929 01:00:33.924098 5345 raft_consensus.cc:3037] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.924121 5210 raft_consensus.cc:3037] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:33.926445 5345 raft_consensus.cc:2454] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 1.
I20260929 01:00:33.926966 4995 leader_election.cc:302] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 5294096528d04a1b858c00d34f0f089b; no voters:
I20260929 01:00:33.927381 5422 raft_consensus.cc:2781] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:33.927647 5210 raft_consensus.cc:2454] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 1.
I20260929 01:00:33.927975 5422 raft_consensus.cc:686] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 1 LEADER]: Becoming Leader. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Running, Role: LEADER
I20260929 01:00:33.928407 5422 consensus_queue.cc:237] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:33.932427 4906 catalog_manager.cc:5495] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193). New cstate: current_term: 1 leader_uuid: "1f5655a6533d4f2283d68fe903bb371b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:34.013042 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.005642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.3.21.193:36547)
0929 01:00:34.006706 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:34.006711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:34.006738 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:00:34.006831 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:34.006834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:34.006914 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:34.007204 (+ 290us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:34.007664 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.008824 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.010384 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.010942 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.011090 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:34.011595 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:34.012281 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:34.012439 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:34.012556 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 01:00:34.012642 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":564,"threads_started":1}
I20260929 01:00:34.037292 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.025418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43797 (local address 127.3.21.193:56129)
0929 01:00:34.031349 (+ 5931us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:34.031370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:34.031580 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:34.031982 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:34.031987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:34.032009 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:34.032368 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:34.032377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.033484 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.033492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:34.035057 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:34.035071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.035920 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.035928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:34.036099 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:34.036162 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:34.036605 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:34.036618 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:00:34.036724 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 01:00:34.036894 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5733,"thread_start_us":5321,"threads_started":1}
I20260929 01:00:34.038024 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.025593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:56129 (local address 127.3.21.194:43797)
0929 01:00:34.025825 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:34.025836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:34.031500 (+ 5664us) server_negotiation.cc:407] Connection header received
0929 01:00:34.031729 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:34.031735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:34.031844 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:34.031974 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:34.032536 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.033286 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.035233 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.035714 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.035868 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:34.036357 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:34.036497 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:34.036605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:34.036744 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 01:00:34.036839 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:00:34.047799 5345 raft_consensus.cc:1261] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Refusing update from remote peer 1f5655a6533d4f2283d68fe903bb371b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:34.048120 5210 raft_consensus.cc:1261] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Refusing update from remote peer 1f5655a6533d4f2283d68fe903bb371b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:34.048604 5468 consensus_queue.cc:1035] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:34.049029 5422 consensus_queue.cc:1035] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:34.055851 5443 mvcc.cc:205] Tried to move back new op lower bound from 7334476325053399040 to 7334476324574838784. Current Snapshot: MvccSnapshot[applied={T|T < 7334476325053399040}]
I20260929 01:00:34.058791 5442 mvcc.cc:205] Tried to move back new op lower bound from 7334476325053399040 to 7334476324574838784. Current Snapshot: MvccSnapshot[applied={T|T < 7334476325053399040}]
I20260929 01:00:34.087283 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.080151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:33697 (local address 127.3.21.193:36547)
0929 01:00:34.080330 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:34.080337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:34.080489 (+ 152us) server_negotiation.cc:407] Connection header received
0929 01:00:34.080686 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:34.080691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:34.080789 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:34.080883 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:34.081529 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.082678 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.085160 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.085699 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.085834 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:34.086444 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:34.086628 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:34.086699 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:34.086952 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 01:00:34.087050 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:00:34.087510 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:34.079709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36547 (local address 127.3.21.194:33697)
0929 01:00:34.080357 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:34.080383 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:34.080534 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:34.081067 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:34.081072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:34.081085 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:00:34.081345 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:34.081357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.082857 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.082861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:34.084439 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:34.084457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:34.085885 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:34.085893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:34.086151 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:34.086209 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:00:34.086789 (+ 580us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:00:34.086800 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:00:34.086903 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:00:34.087100 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":1130,"threads_started":1}
W20260929 01:00:34.088795 5442 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 01:00:35.711582 4906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:35.711817 4906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:35.716297 4906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Sending DeleteTablet for 3 replicas of tablet d722342ce80e401c8b7cbb9a2fc06d1a
I20260929 01:00:35.717249 5045 tablet_service.cc:1514] Processing DeleteTablet for tablet d722342ce80e401c8b7cbb9a2fc06d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:35 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49228
I20260929 01:00:35.717365 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet d722342ce80e401c8b7cbb9a2fc06d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:35 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33970
I20260929 01:00:35.717597 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet d722342ce80e401c8b7cbb9a2fc06d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:35 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48998
I20260929 01:00:35.717960 5504 tablet_replica.cc:330] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: stopping tablet replica
I20260929 01:00:35.718163 5504 raft_consensus.cc:2229] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:35.718789 5504 raft_consensus.cc:2258] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:35.719921 5504 ts_tablet_manager.cc:1905] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:35.727667 5504 ts_tablet_manager.cc:1918] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:00:35.727808 5504 log.cc:1196] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/d722342ce80e401c8b7cbb9a2fc06d1a
I20260929 01:00:35.728202 5504 ts_tablet_manager.cc:1939] T d722342ce80e401c8b7cbb9a2fc06d1a P 1f5655a6533d4f2283d68fe903bb371b: Deleting consensus metadata
I20260929 01:00:35.729470 4895 catalog_manager.cc:4841] TS 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547): tablet d722342ce80e401c8b7cbb9a2fc06d1a (table test-table [id=2b1bae02308d431e94810ecf3969ce5c]) successfully deleted
I20260929 01:00:35.732627 5505 tablet_replica.cc:330] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: stopping tablet replica
I20260929 01:00:35.732877 5505 raft_consensus.cc:2229] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:35.732923 5506 tablet_replica.cc:330] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: stopping tablet replica
I20260929 01:00:35.733127 5505 raft_consensus.cc:2258] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:35.733156 5506 raft_consensus.cc:2229] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:35.733438 5506 raft_consensus.cc:2258] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:35.734769 5505 ts_tablet_manager.cc:1905] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:35.734805 5506 ts_tablet_manager.cc:1905] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:35.742944 5505 ts_tablet_manager.cc:1918] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:00:35.742955 5506 ts_tablet_manager.cc:1918] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:00:35.743124 5506 log.cc:1196] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/d722342ce80e401c8b7cbb9a2fc06d1a
I20260929 01:00:35.743121 5505 log.cc:1196] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/d722342ce80e401c8b7cbb9a2fc06d1a
I20260929 01:00:35.743626 5505 ts_tablet_manager.cc:1939] T d722342ce80e401c8b7cbb9a2fc06d1a P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting consensus metadata
I20260929 01:00:35.743620 5506 ts_tablet_manager.cc:1939] T d722342ce80e401c8b7cbb9a2fc06d1a P 5294096528d04a1b858c00d34f0f089b: Deleting consensus metadata
I20260929 01:00:35.744895 4894 catalog_manager.cc:4841] TS 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797): tablet d722342ce80e401c8b7cbb9a2fc06d1a (table test-table [id=2b1bae02308d431e94810ecf3969ce5c]) successfully deleted
I20260929 01:00:35.744899 4892 catalog_manager.cc:4841] TS 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331): tablet d722342ce80e401c8b7cbb9a2fc06d1a (table test-table [id=2b1bae02308d431e94810ecf3969ce5c]) successfully deleted
Sep 29 01:00:35 dist-test-slave-k0b7 krb5kdc[4863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:00:35 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:35.795698 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:35.772277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.3.21.254:36599)
0929 01:00:35.773745 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:35.773750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:35.773800 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:00:35.773870 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:35.773893 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:35.773982 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:35.774119 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:35.776226 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:35.777361 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:35.779934 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:35.780733 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:35.780856 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:35.786952 (+ 6096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:35.786960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:35.786965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:35.787043 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:35.791779 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:35.792969 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:35.792975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:35.792979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:35.793170 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:35.793798 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:35.793804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:35.793807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:35.794196 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:35.794468 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:35.794941 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 01:00:35.795205 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1351,"thread_start_us":1157,"threads_started":1}
I20260929 01:00:35.801170 4906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:35.802131 4906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:35.810066 5045 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b215e26e64b379def38afa2500370 (DEFAULT_TABLE table=test-table [id=81eaeda3b29b4a198aaaab244ae80193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:35.810268 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b215e26e64b379def38afa2500370 (DEFAULT_TABLE table=test-table [id=81eaeda3b29b4a198aaaab244ae80193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:35.810312 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b215e26e64b379def38afa2500370 (DEFAULT_TABLE table=test-table [id=81eaeda3b29b4a198aaaab244ae80193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:35.810647 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b215e26e64b379def38afa2500370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:35.810709 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b215e26e64b379def38afa2500370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:35.810870 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b215e26e64b379def38afa2500370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:35.817400 5515 tablet_bootstrap.cc:492] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Bootstrap starting.
I20260929 01:00:35.819375 5515 tablet_bootstrap.cc:654] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:35.824491 5515 tablet_bootstrap.cc:492] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: No bootstrap required, opened a new log
I20260929 01:00:35.824703 5515 ts_tablet_manager.cc:1397] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 01:00:35.825526 5516 tablet_bootstrap.cc:492] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Bootstrap starting.
I20260929 01:00:35.825495 5515 raft_consensus.cc:348] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.825769 5515 raft_consensus.cc:374] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:35.825870 5515 raft_consensus.cc:729] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5294096528d04a1b858c00d34f0f089b, State: Initialized, Role: FOLLOWER
I20260929 01:00:35.826258 5515 consensus_queue.cc:260] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.826491 5517 tablet_bootstrap.cc:492] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Bootstrap starting.
I20260929 01:00:35.828095 5517 tablet_bootstrap.cc:654] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:35.829353 5515 ts_tablet_manager.cc:1428] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260929 01:00:35.830588 5516 tablet_bootstrap.cc:654] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:35.832043 5517 tablet_bootstrap.cc:492] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: No bootstrap required, opened a new log
I20260929 01:00:35.832237 5517 ts_tablet_manager.cc:1397] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 01:00:35.833076 5517 raft_consensus.cc:348] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.833328 5517 raft_consensus.cc:374] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:35.833427 5517 raft_consensus.cc:729] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Initialized, Role: FOLLOWER
I20260929 01:00:35.833631 5516 tablet_bootstrap.cc:492] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: No bootstrap required, opened a new log
I20260929 01:00:35.833786 5516 ts_tablet_manager.cc:1397] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260929 01:00:35.833688 5517 consensus_queue.cc:260] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.834825 5517 ts_tablet_manager.cc:1428] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 01:00:35.834873 5516 raft_consensus.cc:348] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.835083 5516 raft_consensus.cc:374] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:35.835155 5516 raft_consensus.cc:729] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Initialized, Role: FOLLOWER
I20260929 01:00:35.835520 5516 consensus_queue.cc:260] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.836048 5516 ts_tablet_manager.cc:1428] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:00:35.849951 5495 raft_consensus.cc:482] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:35.850165 5495 raft_consensus.cc:504] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.851130 5495 leader_election.cc:288] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:35.851862 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b215e26e64b379def38afa2500370" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b" is_pre_election: true
I20260929 01:00:35.851907 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b215e26e64b379def38afa2500370" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" is_pre_election: true
I20260929 01:00:35.852128 5345 raft_consensus.cc:2454] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 0.
I20260929 01:00:35.852270 5210 raft_consensus.cc:2454] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 0.
I20260929 01:00:35.852669 4995 leader_election.cc:302] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 5294096528d04a1b858c00d34f0f089b; no voters:
I20260929 01:00:35.853003 5495 raft_consensus.cc:2781] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:35.853153 5495 raft_consensus.cc:482] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:35.853282 5495 raft_consensus.cc:3037] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:35.855556 5495 raft_consensus.cc:504] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.856192 5495 leader_election.cc:288] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 election: Requested vote from peers 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:35.856768 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b215e26e64b379def38afa2500370" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b"
I20260929 01:00:35.856834 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b215e26e64b379def38afa2500370" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b"
I20260929 01:00:35.856986 5345 raft_consensus.cc:3037] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:35.857112 5210 raft_consensus.cc:3037] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:35.859155 5345 raft_consensus.cc:2454] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 1.
I20260929 01:00:35.859274 5210 raft_consensus.cc:2454] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 1.
I20260929 01:00:35.859766 4997 leader_election.cc:302] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:35.860095 5495 raft_consensus.cc:2781] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:35.860265 5495 raft_consensus.cc:686] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 1 LEADER]: Becoming Leader. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Running, Role: LEADER
I20260929 01:00:35.860541 5495 consensus_queue.cc:237] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.863415 4907 catalog_manager.cc:5495] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193). New cstate: current_term: 1 leader_uuid: "1f5655a6533d4f2283d68fe903bb371b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:35.895514 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:35.889593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.3.21.193:36547)
0929 01:00:35.890366 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:35.890371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:35.890392 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:00:35.890453 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:35.890457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:35.890615 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:35.890713 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:35.891127 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:35.892060 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:35.893426 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:35.893933 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:35.894038 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:35.894463 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:35.894640 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:35.894793 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:35.895009 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 01:00:35.895115 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":490,"threads_started":1}
I20260929 01:00:35.906945 5210 raft_consensus.cc:1261] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Refusing update from remote peer 1f5655a6533d4f2283d68fe903bb371b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:35.907013 5345 raft_consensus.cc:1261] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Refusing update from remote peer 1f5655a6533d4f2283d68fe903bb371b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:35.907585 5499 consensus_queue.cc:1035] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:35.907939 5495 consensus_queue.cc:1035] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:35.958443 4907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:35.958863 4907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:35.962248 4907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Sending DeleteTablet for 3 replicas of tablet 1c0b215e26e64b379def38afa2500370
I20260929 01:00:35.963361 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b215e26e64b379def38afa2500370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:35 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33970
I20260929 01:00:35.963382 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b215e26e64b379def38afa2500370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:35 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48998
I20260929 01:00:35.963354 5045 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b215e26e64b379def38afa2500370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:35 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49228
I20260929 01:00:35.963701 5506 tablet_replica.cc:330] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: stopping tablet replica
I20260929 01:00:35.963711 5504 tablet_replica.cc:330] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: stopping tablet replica
I20260929 01:00:35.963876 5506 raft_consensus.cc:2229] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:35.963898 5504 raft_consensus.cc:2229] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:35.964097 5505 tablet_replica.cc:330] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: stopping tablet replica
I20260929 01:00:35.964118 5506 raft_consensus.cc:2258] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:35.964234 5504 raft_consensus.cc:2258] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:35.964316 5505 raft_consensus.cc:2229] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:35.964514 5505 raft_consensus.cc:2258] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:35.965453 5504 ts_tablet_manager.cc:1905] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:35.965811 5506 ts_tablet_manager.cc:1905] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:35.965842 5505 ts_tablet_manager.cc:1905] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:35.964913 4907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:35.966836 4907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:35.972934 5504 ts_tablet_manager.cc:1918] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:35.973146 5504 log.cc:1196] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/1c0b215e26e64b379def38afa2500370
I20260929 01:00:35.973385 5506 ts_tablet_manager.cc:1918] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:35.973519 5505 ts_tablet_manager.cc:1918] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:00:35.973577 5504 ts_tablet_manager.cc:1939] T 1c0b215e26e64b379def38afa2500370 P 1f5655a6533d4f2283d68fe903bb371b: Deleting consensus metadata
I20260929 01:00:35.974012 5506 log.cc:1196] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/1c0b215e26e64b379def38afa2500370
I20260929 01:00:35.974077 5505 log.cc:1196] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/1c0b215e26e64b379def38afa2500370
I20260929 01:00:35.974514 5506 ts_tablet_manager.cc:1939] T 1c0b215e26e64b379def38afa2500370 P 5294096528d04a1b858c00d34f0f089b: Deleting consensus metadata
I20260929 01:00:35.974529 5505 ts_tablet_manager.cc:1939] T 1c0b215e26e64b379def38afa2500370 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting consensus metadata
I20260929 01:00:35.975126 4895 catalog_manager.cc:4841] TS 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547): tablet 1c0b215e26e64b379def38afa2500370 (table test-table [id=81eaeda3b29b4a198aaaab244ae80193]) successfully deleted
I20260929 01:00:35.975670 4894 catalog_manager.cc:4841] TS 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797): tablet 1c0b215e26e64b379def38afa2500370 (table test-table [id=81eaeda3b29b4a198aaaab244ae80193]) successfully deleted
I20260929 01:00:35.975661 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 661f4391f3ee42548283b92446e1cd52 (DEFAULT_TABLE table=test-table [id=c3f3a2afa1394863849e1f79c7c3ecd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:35.975805 5045 tablet_service.cc:1467] Processing CreateTablet for tablet 661f4391f3ee42548283b92446e1cd52 (DEFAULT_TABLE table=test-table [id=c3f3a2afa1394863849e1f79c7c3ecd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:35.976127 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 661f4391f3ee42548283b92446e1cd52 (DEFAULT_TABLE table=test-table [id=c3f3a2afa1394863849e1f79c7c3ecd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:35.976225 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661f4391f3ee42548283b92446e1cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:35.976346 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661f4391f3ee42548283b92446e1cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:35.976492 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661f4391f3ee42548283b92446e1cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:35.979118 4892 catalog_manager.cc:4841] TS 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331): tablet 1c0b215e26e64b379def38afa2500370 (table test-table [id=81eaeda3b29b4a198aaaab244ae80193]) successfully deleted
I20260929 01:00:35.982244 5517 tablet_bootstrap.cc:492] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Bootstrap starting.
I20260929 01:00:35.983506 5516 tablet_bootstrap.cc:492] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Bootstrap starting.
I20260929 01:00:35.984272 5517 tablet_bootstrap.cc:654] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:35.985685 5516 tablet_bootstrap.cc:654] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:35.985801 5515 tablet_bootstrap.cc:492] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Bootstrap starting.
I20260929 01:00:35.987402 5517 tablet_bootstrap.cc:492] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: No bootstrap required, opened a new log
I20260929 01:00:35.987679 5517 ts_tablet_manager.cc:1397] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:00:35.987758 5515 tablet_bootstrap.cc:654] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:35.988859 5517 raft_consensus.cc:348] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.989197 5517 raft_consensus.cc:374] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:35.989332 5517 raft_consensus.cc:729] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7976b7a0cb4b01891c9e51b7f1739b, State: Initialized, Role: FOLLOWER
I20260929 01:00:35.989675 5517 consensus_queue.cc:260] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.990448 5515 tablet_bootstrap.cc:492] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: No bootstrap required, opened a new log
I20260929 01:00:35.990535 5516 tablet_bootstrap.cc:492] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: No bootstrap required, opened a new log
I20260929 01:00:35.990656 5515 ts_tablet_manager.cc:1397] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:00:35.990713 5516 ts_tablet_manager.cc:1397] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260929 01:00:35.991604 5515 raft_consensus.cc:348] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.991792 5515 raft_consensus.cc:374] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:35.991842 5515 raft_consensus.cc:729] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5294096528d04a1b858c00d34f0f089b, State: Initialized, Role: FOLLOWER
I20260929 01:00:35.991740 5516 raft_consensus.cc:348] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.991992 5516 raft_consensus.cc:374] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:35.992079 5516 raft_consensus.cc:729] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Initialized, Role: FOLLOWER
I20260929 01:00:35.992096 5515 consensus_queue.cc:260] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.992348 5516 consensus_queue.cc:260] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:35.992693 5515 ts_tablet_manager.cc:1428] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 01:00:35.992928 5516 ts_tablet_manager.cc:1428] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 01:00:35.993643 5517 ts_tablet_manager.cc:1428] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260929 01:00:36.219084 5519 raft_consensus.cc:482] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:36.219477 5519 raft_consensus.cc:504] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:36.221189 5519 leader_election.cc:288] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547), 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797)
I20260929 01:00:36.224879 5528 raft_consensus.cc:482] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:36.225098 5528 raft_consensus.cc:504] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:36.225986 5528 leader_election.cc:288] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:36.226539 5209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661f4391f3ee42548283b92446e1cd52" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" is_pre_election: true
I20260929 01:00:36.226838 5209 raft_consensus.cc:2454] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 0.
I20260929 01:00:36.226787 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661f4391f3ee42548283b92446e1cd52" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b" is_pre_election: true
I20260929 01:00:36.227111 5345 raft_consensus.cc:2454] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 0.
I20260929 01:00:36.227401 4997 leader_election.cc:302] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:36.227723 5528 raft_consensus.cc:2781] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:36.227840 5528 raft_consensus.cc:482] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:36.227923 5528 raft_consensus.cc:3037] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:36.231722 5528 raft_consensus.cc:504] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:36.232267 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.221546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36547 (local address 127.3.21.195:58561)
0929 01:00:36.222938 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:36.222971 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:36.223169 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:36.223567 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:36.223573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:36.223606 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:36.224093 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:36.224118 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.226055 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.226059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:36.229113 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:36.229131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.231106 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.231114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:36.231360 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:36.231389 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:00:36.231510 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 01:00:36.231679 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":454,"spinlock_wait_cycles":22144,"thread_start_us":1135,"threads_started":1}
I20260929 01:00:36.232823 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.221650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:58561 (local address 127.3.21.193:36547)
0929 01:00:36.221891 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:36.221902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:36.223088 (+ 1186us) server_negotiation.cc:407] Connection header received
0929 01:00:36.223317 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:36.223323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:36.223402 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:36.223513 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:36.224483 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.225838 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.229320 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.230873 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.231088 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:36.231188 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:36.231548 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:00:36.231692 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:00:36.232918 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.221574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43797 (local address 127.3.21.195:35005)
0929 01:00:36.222969 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:36.222986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:36.223161 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:36.223581 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:36.223585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:36.223612 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:36.224396 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:36.224412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.226053 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.226059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:36.229434 (+ 3375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:36.229453 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.231914 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.231922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:36.232120 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:36.232179 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 01:00:36.232341 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 01:00:36.232501 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"mutex_wait_us":599,"spinlock_wait_cycles":28544,"thread_start_us":999,"threads_started":1}
I20260929 01:00:36.233094 5209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661f4391f3ee42548283b92446e1cd52" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b"
I20260929 01:00:36.233386 5209 raft_consensus.cc:3037] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:36.233388 5064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661f4391f3ee42548283b92446e1cd52" candidate_uuid: "5294096528d04a1b858c00d34f0f089b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5655a6533d4f2283d68fe903bb371b" is_pre_election: true
I20260929 01:00:36.233733 5064 raft_consensus.cc:2379] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5294096528d04a1b858c00d34f0f089b in current term 1: Already voted for candidate 1f5655a6533d4f2283d68fe903bb371b in this term.
I20260929 01:00:36.234046 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:36.221650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:35005 (local address 127.3.21.194:43797)
0929 01:00:36.222429 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:36.222436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:36.223088 (+ 652us) server_negotiation.cc:407] Connection header received
0929 01:00:36.223317 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:36.223323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:36.223402 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:36.223519 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:36.224631 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.225836 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.229626 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:36.231471 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:36.231650 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:36.231814 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:36.233114 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0929 01:00:36.233344 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":431,"threads_started":1}
I20260929 01:00:36.234215 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661f4391f3ee42548283b92446e1cd52" candidate_uuid: "1f5655a6533d4f2283d68fe903bb371b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5294096528d04a1b858c00d34f0f089b"
I20260929 01:00:36.234582 5345 raft_consensus.cc:3037] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:36.233728 5528 leader_election.cc:288] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 election: Requested vote from peers 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797), 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331)
I20260929 01:00:36.235920 5210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661f4391f3ee42548283b92446e1cd52" candidate_uuid: "5294096528d04a1b858c00d34f0f089b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" is_pre_election: true
I20260929 01:00:36.236678 5277 leader_election.cc:302] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5294096528d04a1b858c00d34f0f089b; no voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b
I20260929 01:00:36.237165 5209 raft_consensus.cc:2454] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 1.
I20260929 01:00:36.237604 4997 leader_election.cc:302] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5655a6533d4f2283d68fe903bb371b, 7a7976b7a0cb4b01891c9e51b7f1739b; no voters:
I20260929 01:00:36.237910 5528 raft_consensus.cc:2781] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:36.238066 5528 raft_consensus.cc:686] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 1 LEADER]: Becoming Leader. State: Replica: 1f5655a6533d4f2283d68fe903bb371b, State: Running, Role: LEADER
I20260929 01:00:36.238047 5345 raft_consensus.cc:2454] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5655a6533d4f2283d68fe903bb371b in term 1.
I20260929 01:00:36.238221 5519 raft_consensus.cc:2726] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:00:36.238399 5528 consensus_queue.cc:237] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } }
I20260929 01:00:36.241488 4907 catalog_manager.cc:5495] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193). New cstate: current_term: 1 leader_uuid: "1f5655a6533d4f2283d68fe903bb371b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5655a6533d4f2283d68fe903bb371b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:36.308974 5345 raft_consensus.cc:1261] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Refusing update from remote peer 1f5655a6533d4f2283d68fe903bb371b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:36.309440 5209 raft_consensus.cc:1261] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Refusing update from remote peer 1f5655a6533d4f2283d68fe903bb371b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:36.309617 5528 consensus_queue.cc:1035] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294096528d04a1b858c00d34f0f089b" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:36.310179 5499 consensus_queue.cc:1035] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7976b7a0cb4b01891c9e51b7f1739b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:00:36.335016 5486 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 01:00:37.928944 4906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:37.929303 4906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:37.934729 4906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64a625778cd3491c8e005c9146d44aad: Sending DeleteTablet for 3 replicas of tablet 661f4391f3ee42548283b92446e1cd52
I20260929 01:00:37.935716 5045 tablet_service.cc:1514] Processing DeleteTablet for tablet 661f4391f3ee42548283b92446e1cd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:37 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49228
I20260929 01:00:37.936033 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 661f4391f3ee42548283b92446e1cd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:37 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48998
I20260929 01:00:37.936637 5560 tablet_replica.cc:330] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: stopping tablet replica
I20260929 01:00:37.936650 5559 tablet_replica.cc:330] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: stopping tablet replica
I20260929 01:00:37.936862 5560 raft_consensus.cc:2229] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:37.936851 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet 661f4391f3ee42548283b92446e1cd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:37 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33970
I20260929 01:00:37.937060 5559 raft_consensus.cc:2229] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:37.937124 5560 raft_consensus.cc:2258] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:37.937534 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4979
I20260929 01:00:37.938381 5560 ts_tablet_manager.cc:1905] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:37.938762 5559 raft_consensus.cc:2258] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:37.939868 5559 ts_tablet_manager.cc:1905] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:37.947045 5560 ts_tablet_manager.cc:1918] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:00:37.947248 5560 log.cc:1196] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/661f4391f3ee42548283b92446e1cd52
I20260929 01:00:37.947868 5560 ts_tablet_manager.cc:1939] T 661f4391f3ee42548283b92446e1cd52 P 5294096528d04a1b858c00d34f0f089b: Deleting consensus metadata
I20260929 01:00:37.949265 5559 ts_tablet_manager.cc:1918] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:00:37.949318 4892 catalog_manager.cc:4841] TS 5294096528d04a1b858c00d34f0f089b (127.3.21.195:36331): tablet 661f4391f3ee42548283b92446e1cd52 (table test-table [id=c3f3a2afa1394863849e1f79c7c3ecd1]) successfully deleted
I20260929 01:00:37.949659 5559 log.cc:1196] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/661f4391f3ee42548283b92446e1cd52
I20260929 01:00:37.950284 5559 ts_tablet_manager.cc:1939] T 661f4391f3ee42548283b92446e1cd52 P 1f5655a6533d4f2283d68fe903bb371b: Deleting consensus metadata
I20260929 01:00:37.951723 4895 catalog_manager.cc:4841] TS 1f5655a6533d4f2283d68fe903bb371b (127.3.21.193:36547): tablet 661f4391f3ee42548283b92446e1cd52 (table test-table [id=c3f3a2afa1394863849e1f79c7c3ecd1]) successfully deleted
I20260929 01:00:37.966408 5561 tablet_replica.cc:330] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: stopping tablet replica
I20260929 01:00:37.966782 5561 raft_consensus.cc:2229] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:37.967177 5561 raft_consensus.cc:2258] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:37.968371 5561 ts_tablet_manager.cc:1905] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 01:00:37 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:37.976938 5561 ts_tablet_manager.cc:1918] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:00:37.977115 5561 log.cc:1196] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/661f4391f3ee42548283b92446e1cd52
I20260929 01:00:37.977674 5561 ts_tablet_manager.cc:1939] T 661f4391f3ee42548283b92446e1cd52 P 7a7976b7a0cb4b01891c9e51b7f1739b: Deleting consensus metadata
I20260929 01:00:37.979008 4894 catalog_manager.cc:4841] TS 7a7976b7a0cb4b01891c9e51b7f1739b (127.3.21.194:43797): tablet 661f4391f3ee42548283b92446e1cd52 (table test-table [id=c3f3a2afa1394863849e1f79c7c3ecd1]) successfully deleted
I20260929 01:00:37.983464 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:37.941893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.3.21.193:36547)
0929 01:00:37.964777 (+ 22884us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:37.964787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:37.964814 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:00:37.964891 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:37.964900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:37.965004 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:37.965157 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:37.966976 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:37.967987 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:37.969991 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:37.970747 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:37.970898 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:37.976560 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:37.976579 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:37.976595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:37.976638 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:37.980407 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:37.981138 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:37.981148 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:37.981152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:37.981302 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:37.981708 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:37.981712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:37.981716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:37.982086 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:37.982559 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:37.982881 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 01:00:37.983101 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22722,"thread_start_us":23441,"threads_started":1}
W20260929 01:00:37.984243 5110 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49266
W20260929 01:00:37.984889 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:37.985237 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4979
I20260929 01:00:38.009759 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5119
Sep 29 01:00:38 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:38.027727 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:38.010730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.3.21.194:43797)
0929 01:00:38.011259 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:38.011265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:38.011282 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:00:38.011336 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:38.011340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:38.011408 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:38.011513 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:38.012942 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:38.013772 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:38.015512 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:38.015956 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:38.016066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:38.020823 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:38.020841 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:38.020858 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:38.020899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:38.024251 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:38.025207 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:38.025215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:38.025218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:38.025366 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:38.025829 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:38.025833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:38.025835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:38.026448 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:38.026769 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:38.027083 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 01:00:38.027347 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
W20260929 01:00:38.028412 5250 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34010
W20260929 01:00:38.029136 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5119: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:38.029417 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 5119
I20260929 01:00:38.057556 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5259
Sep 29 01:00:38 dist-test-slave-k0b7 krb5kdc[4863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643635, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:38.076283 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:38.058441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.3.21.195:36331)
0929 01:00:38.059032 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:38.059037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:38.059055 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:00:38.059123 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:38.059127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:38.059190 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:38.059391 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:38.060713 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:38.061611 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:38.062983 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:38.063528 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:38.063639 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:38.068658 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:38.068719 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:38.068746 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:38.068797 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:38.072834 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:38.073725 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:38.073734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:38.073737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:38.073941 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:38.074408 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:38.074412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:38.074414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:38.074976 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:38.075345 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:38.075572 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 01:00:38.075779 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":323,"threads_started":1}
W20260929 01:00:38.077291 5390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49002
W20260929 01:00:38.078199 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5259: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:38.078384 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 5259
I20260929 01:00:38.097527 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 4876
I20260929 01:00:38.098809 4948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:38.217780 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 4876
2026-09-29T01:00:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (11849 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:38 dist-test-slave-k0b7 krb5kdc[5569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:38 dist-test-slave-k0b7 krb5kdc[5569](info): set up 2 sockets
Sep 29 01:00:38 dist-test-slave-k0b7 krb5kdc[5569](info): commencing operation
krb5kdc: starting...
W20260929 01:00:39.371574 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:39 dist-test-slave-k0b7 krb5kdc[5569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:39.608472 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:45219
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:39.956393 5582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:39.956717 5582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:39.956878 5582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:39.964419 5582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:39.964543 5582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:39.964639 5582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:39.964735 5582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:39.964818 5582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:39.975049 5582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:45219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:39.976573 5582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:39.977914 5582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:39.991767 5587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:39.991828 5590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:39.991750 5588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:39.992079 5582 server_base.cc:1034] running on GCE node
I20260929 01:00:39.993245 5582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:39.994851 5582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:39.996138 5582 hybrid_clock.cc:648] HybridClock initialized: now 1790643639996081 us; error 51 us; skew 500 ppm
I20260929 01:00:39.996512 5582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:39 dist-test-slave-k0b7 krb5kdc[5569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643639, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:39.999972 5582 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:40.001972 5582 webserver.cc:460] Webserver started at http://127.3.21.254:36107/ using document root <none> and password file <none>
I20260929 01:00:40.002821 5582 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:40.002982 5582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:40.003337 5582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:40.006680 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "14814ae7bd664cd0a305a1ca63f20194"
format_stamp: "Formatted at 2026-09-29 01:00:40 on dist-test-slave-k0b7"
server_key: "cce5bd56f38abcf1cd4e19c5bca744c3"
server_key_iv: "37d2440103d40a408d250c1087707cda"
server_key_version: "encryptionkey@0"
I20260929 01:00:40.007565 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "14814ae7bd664cd0a305a1ca63f20194"
format_stamp: "Formatted at 2026-09-29 01:00:40 on dist-test-slave-k0b7"
server_key: "cce5bd56f38abcf1cd4e19c5bca744c3"
server_key_iv: "37d2440103d40a408d250c1087707cda"
server_key_version: "encryptionkey@0"
I20260929 01:00:40.013463 5582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:00:40.017397 5597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:40.019273 5582 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:40.019488 5582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "14814ae7bd664cd0a305a1ca63f20194"
format_stamp: "Formatted at 2026-09-29 01:00:40 on dist-test-slave-k0b7"
server_key: "cce5bd56f38abcf1cd4e19c5bca744c3"
server_key_iv: "37d2440103d40a408d250c1087707cda"
server_key_version: "encryptionkey@0"
I20260929 01:00:40.019711 5582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:40.038894 5582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:40.044373 5582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:40.044714 5582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:40.065675 5582 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:45219
I20260929 01:00:40.065721 5648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:45219 every 8 connection(s)
I20260929 01:00:40.067204 5582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:40.070437 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 5582
I20260929 01:00:40.070729 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:40.071339 5649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:40.071352 3159 external_mini_cluster.cc:1428] Setting key e6cf977cd9a096dbe76433ef968d6ee9
I20260929 01:00:40.087805 5649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194: Bootstrap starting.
Sep 29 01:00:40 dist-test-slave-k0b7 krb5kdc[5569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:40.091609 5649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:40.092861 5649 log.cc:824] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:40.096072 5649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194: No bootstrap required, opened a new log
I20260929 01:00:40.097507 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:40.073549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.3.21.254:45219)
0929 01:00:40.074561 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:40.074575 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:40.074625 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:00:40.076529 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:40.076555 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:40.077033 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:40.077419 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:40.079653 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:40.081132 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:40.083302 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:40.084200 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:40.084369 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:40.089851 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:40.089861 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:40.089879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:40.089934 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:40.093242 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:40.094076 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:40.094083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:40.094090 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:40.094282 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:40.094760 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:40.094764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:40.094766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:40.095291 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:40.095614 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:40.096045 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 01:00:40.096509 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":411,"threads_started":1}
I20260929 01:00:40.104415 5649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14814ae7bd664cd0a305a1ca63f20194" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45219 } }
I20260929 01:00:40.104864 5649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:40.105121 5649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14814ae7bd664cd0a305a1ca63f20194, State: Initialized, Role: FOLLOWER
I20260929 01:00:40.105726 5649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14814ae7bd664cd0a305a1ca63f20194" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45219 } }
I20260929 01:00:40.105975 5649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:40.106172 5649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:40.106353 5649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:40.108629 5649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14814ae7bd664cd0a305a1ca63f20194" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45219 } }
I20260929 01:00:40.109179 5649 leader_election.cc:302] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14814ae7bd664cd0a305a1ca63f20194; no voters:
I20260929 01:00:40.109851 5649 leader_election.cc:288] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:40.109994 5654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:40.110709 5654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [term 1 LEADER]: Becoming Leader. State: Replica: 14814ae7bd664cd0a305a1ca63f20194, State: Running, Role: LEADER
I20260929 01:00:40.111248 5654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14814ae7bd664cd0a305a1ca63f20194" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45219 } }
I20260929 01:00:40.112183 5649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:40.114379 5655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14814ae7bd664cd0a305a1ca63f20194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14814ae7bd664cd0a305a1ca63f20194" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45219 } } }
I20260929 01:00:40.114260 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14814ae7bd664cd0a305a1ca63f20194. Latest consensus state: current_term: 1 leader_uuid: "14814ae7bd664cd0a305a1ca63f20194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14814ae7bd664cd0a305a1ca63f20194" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45219 } } }
I20260929 01:00:40.114769 5655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:40.114807 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:40.115746 5661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:40.121840 5661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:40.132046 5661 catalog_manager.cc:1347] Generated new cluster ID: 2c788e9f3b034b649355c8031f86da0c
I20260929 01:00:40.132193 5661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:40.199776 5661 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:40.201258 5661 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:40.210031 5661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14814ae7bd664cd0a305a1ca63f20194: Generated new TSK 0
I20260929 01:00:40.211201 5661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:40.279321 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45219
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:40.635571 5675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:40.635895 5675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:40.635993 5675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:40.643811 5675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:40.643949 5675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:40.644064 5675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:40.654543 5675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:40.655866 5675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:40.657140 5675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:40.671555 5683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:40.671588 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:40.671628 5681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:40.672335 5675 server_base.cc:1034] running on GCE node
I20260929 01:00:40.672922 5675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:40.673925 5675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:40.675181 5675 hybrid_clock.cc:648] HybridClock initialized: now 1790643640675143 us; error 39 us; skew 500 ppm
I20260929 01:00:40.675486 5675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:40 dist-test-slave-k0b7 krb5kdc[5569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643640, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:40.679150 5675 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:40.680970 5675 webserver.cc:460] Webserver started at http://127.3.21.193:44639/ using document root <none> and password file <none>
I20260929 01:00:40.681741 5675 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:40.681876 5675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:40.682233 5675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:40.685437 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "fb2c340b074946beb7b9950fbddcefcc"
format_stamp: "Formatted at 2026-09-29 01:00:40 on dist-test-slave-k0b7"
server_key: "bfab6f36bf331b15e4c8378c36eeeced"
server_key_iv: "807a9c50a0b0623db809b4df5d1806b8"
server_key_version: "encryptionkey@0"
I20260929 01:00:40.686311 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "fb2c340b074946beb7b9950fbddcefcc"
format_stamp: "Formatted at 2026-09-29 01:00:40 on dist-test-slave-k0b7"
server_key: "bfab6f36bf331b15e4c8378c36eeeced"
server_key_iv: "807a9c50a0b0623db809b4df5d1806b8"
server_key_version: "encryptionkey@0"
I20260929 01:00:40.692104 5675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:00:40.696007 5690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:40.697474 5675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:40.697803 5675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "fb2c340b074946beb7b9950fbddcefcc"
format_stamp: "Formatted at 2026-09-29 01:00:40 on dist-test-slave-k0b7"
server_key: "bfab6f36bf331b15e4c8378c36eeeced"
server_key_iv: "807a9c50a0b0623db809b4df5d1806b8"
server_key_version: "encryptionkey@0"
I20260929 01:00:40.698052 5675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:40.733920 5675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:40.738656 5675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:40.738986 5675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:40.740029 5675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:40.741514 5675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:40.741619 5675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:40.741783 5675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:40.741858 5675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:40.781610 5675 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:43329
I20260929 01:00:40.781662 5802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:43329 every 8 connection(s)
I20260929 01:00:40.783043 5675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:40.792518 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 5675
I20260929 01:00:40.792779 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:40.793599 3159 external_mini_cluster.cc:1428] Setting key 9581451c9519313fcee21da61cc4c6c7
Sep 29 01:00:40 dist-test-slave-k0b7 krb5kdc[5569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643640, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:00:40.816581 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:40.785704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:44325 (local address 127.3.21.254:45219)
0929 01:00:40.786383 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:40.786392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:40.787526 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 01:00:40.789395 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:40.789400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:40.789473 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:40.789593 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:40.791328 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:40.792197 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:40.794849 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:40.795452 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:40.795606 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:40.808936 (+ 13330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:40.808947 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:40.808953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:40.809008 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:40.812799 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:40.813927 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:40.813933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:40.813944 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:40.814079 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:40.814577 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:40.814582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:40.814585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:40.814885 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:40.815059 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:40.815915 (+ 856us) server_negotiation.cc:299] Negotiation successful
0929 01:00:40.816121 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":453,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:40.817541 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:40.786287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45219 (local address 127.3.21.193:44325)
0929 01:00:40.787342 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:40.787385 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:40.789091 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:40.789806 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:40.789816 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:40.790467 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:40.791138 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:40.791157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:40.792403 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:40.792410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:40.794672 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:40.794689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:40.795772 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:40.795780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:40.796076 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:40.804676 (+ 8600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:40.804718 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:40.808567 (+ 3849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:40.813144 (+ 4577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:40.813154 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:40.813175 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:40.813771 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:40.814202 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:40.814206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:40.814212 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:40.814417 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:40.815030 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:40.815039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:40.815421 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 01:00:40.815724 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 01:00:40.816202 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":483,"threads_started":1}
I20260929 01:00:40.819448 5803 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45219
I20260929 01:00:40.819823 5803 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:40.820746 5803 heartbeater.cc:511] Master 127.3.21.254:45219 requested a full tablet report, sending...
I20260929 01:00:40.823359 5614 ts_manager.cc:194] Registered new tserver with Master: fb2c340b074946beb7b9950fbddcefcc (127.3.21.193:43329)
I20260929 01:00:40.825657 5614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:44325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:40.847169 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45219
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:41.219879 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:41.220198 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:41.220335 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:41.228235 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:41.228360 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:41.228549 5809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:41.239217 5809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:41.240710 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:41.242162 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:41.257772 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:41.257752 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:41.257793 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:41.258643 5809 server_base.cc:1034] running on GCE node
I20260929 01:00:41.259315 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:41.260373 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:41.261628 5809 hybrid_clock.cc:648] HybridClock initialized: now 1790643641261585 us; error 49 us; skew 500 ppm
I20260929 01:00:41.261976 5809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:41 dist-test-slave-k0b7 krb5kdc[5569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:41.266101 5809 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:41.267843 5809 webserver.cc:460] Webserver started at http://127.3.21.194:38563/ using document root <none> and password file <none>
I20260929 01:00:41.268625 5809 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:41.268775 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:41.269111 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:41.272396 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "1dbf3be05f0b4b68b93fb116d314c0ad"
format_stamp: "Formatted at 2026-09-29 01:00:41 on dist-test-slave-k0b7"
server_key: "6299466922f3cc6ca17dff454787f63a"
server_key_iv: "18ca149d53ab914f37647e1ecbac914d"
server_key_version: "encryptionkey@0"
I20260929 01:00:41.273278 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "1dbf3be05f0b4b68b93fb116d314c0ad"
format_stamp: "Formatted at 2026-09-29 01:00:41 on dist-test-slave-k0b7"
server_key: "6299466922f3cc6ca17dff454787f63a"
server_key_iv: "18ca149d53ab914f37647e1ecbac914d"
server_key_version: "encryptionkey@0"
I20260929 01:00:41.279356 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:00:41.283466 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:41.285331 5809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:41.285565 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "1dbf3be05f0b4b68b93fb116d314c0ad"
format_stamp: "Formatted at 2026-09-29 01:00:41 on dist-test-slave-k0b7"
server_key: "6299466922f3cc6ca17dff454787f63a"
server_key_iv: "18ca149d53ab914f37647e1ecbac914d"
server_key_version: "encryptionkey@0"
I20260929 01:00:41.285851 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:41.310189 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:41.315986 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:41.316401 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:41.317749 5809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:41.319764 5809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:41.319886 5809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:41.320044 5809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:41.320119 5809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:41.364681 5809 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:41065
I20260929 01:00:41.364701 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:41065 every 8 connection(s)
I20260929 01:00:41.366135 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:41.370781 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 5809
I20260929 01:00:41.371026 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:41.371520 3159 external_mini_cluster.cc:1428] Setting key 48b36c4308d9e6468b57d56f6daddc10
Sep 29 01:00:41 dist-test-slave-k0b7 krb5kdc[5569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:00:41.394819 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:41.369411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:39677 (local address 127.3.21.254:45219)
0929 01:00:41.370165 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:41.370170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:41.370986 (+ 816us) server_negotiation.cc:407] Connection header received
0929 01:00:41.373054 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:41.373061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:41.373143 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:41.373278 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:41.375339 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.376808 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.379858 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.380843 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.381001 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:41.385726 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:41.385737 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:41.385745 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:41.385798 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:41.389918 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:41.391133 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:41.391140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:41.391143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:41.391345 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:41.392116 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:41.392126 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:41.392129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:41.392622 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:41.392864 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:41.394015 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0929 01:00:41.394318 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"spinlock_wait_cycles":11648,"thread_start_us":550,"threads_started":1}
I20260929 01:00:41.395830 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:41.369837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45219 (local address 127.3.21.194:39677)
0929 01:00:41.370702 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:41.370756 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:41.372771 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:41.373722 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:41.373732 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:41.374422 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:41.375127 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:41.375147 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.377135 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.377141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:41.379537 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:41.379571 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.381304 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.381317 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:41.381823 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:41.382944 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:41.382976 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:41.385342 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:41.390147 (+ 4805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:41.390156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:41.390173 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:41.390935 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:41.391487 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:41.391495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:41.391501 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:41.391807 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:41.392834 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:41.392845 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:41.393373 (+ 528us) client_negotiation.cc:769] Sending connection context
0929 01:00:41.393720 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 01:00:41.394377 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:00:41.398376 5937 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45219
I20260929 01:00:41.398883 5937 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:41.399724 5937 heartbeater.cc:511] Master 127.3.21.254:45219 requested a full tablet report, sending...
I20260929 01:00:41.401708 5614 ts_manager.cc:194] Registered new tserver with Master: 1dbf3be05f0b4b68b93fb116d314c0ad (127.3.21.194:41065)
I20260929 01:00:41.403409 5614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:39677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:41.432982 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45219
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:41.795740 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:41.796108 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:41.796314 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:41.805313 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:41.805454 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:41.805671 5943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:41.817705 5943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:41.819263 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:41.820710 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:00:41.829325 5803 heartbeater.cc:503] Master 127.3.21.254:45219 was elected leader, sending a full tablet report...
W20260929 01:00:41.836542 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:41.836635 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:41.836611 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:41.837345 5943 server_base.cc:1034] running on GCE node
I20260929 01:00:41.838049 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:41.839299 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:41.840654 5943 hybrid_clock.cc:648] HybridClock initialized: now 1790643641840546 us; error 89 us; skew 500 ppm
I20260929 01:00:41.841032 5943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:41 dist-test-slave-k0b7 krb5kdc[5569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:41.844884 5943 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:41.846591 5943 webserver.cc:460] Webserver started at http://127.3.21.195:42039/ using document root <none> and password file <none>
I20260929 01:00:41.847292 5943 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:41.847394 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:41.847684 5943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:41.850775 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "db2044fe320945968f02718b2a3689cd"
format_stamp: "Formatted at 2026-09-29 01:00:41 on dist-test-slave-k0b7"
server_key: "7ec019ff2ee644888bee281f73fc7d6a"
server_key_iv: "01983ddc132c170507942601f3edb13c"
server_key_version: "encryptionkey@0"
I20260929 01:00:41.851789 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "db2044fe320945968f02718b2a3689cd"
format_stamp: "Formatted at 2026-09-29 01:00:41 on dist-test-slave-k0b7"
server_key: "7ec019ff2ee644888bee281f73fc7d6a"
server_key_iv: "01983ddc132c170507942601f3edb13c"
server_key_version: "encryptionkey@0"
I20260929 01:00:41.858246 5943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:00:41.863036 5958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:41.864812 5943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:00:41.864985 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "db2044fe320945968f02718b2a3689cd"
format_stamp: "Formatted at 2026-09-29 01:00:41 on dist-test-slave-k0b7"
server_key: "7ec019ff2ee644888bee281f73fc7d6a"
server_key_iv: "01983ddc132c170507942601f3edb13c"
server_key_version: "encryptionkey@0"
I20260929 01:00:41.865181 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:41.906625 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:41.912400 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:41.912685 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:41.913815 5943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:41.915434 5943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:41.915532 5943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:41.915623 5943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:41.915659 5943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:41.960534 5943 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:33757
I20260929 01:00:41.960578 6070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:33757 every 8 connection(s)
I20260929 01:00:41.961997 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:41.966717 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 5943
I20260929 01:00:41.966925 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:41.967397 3159 external_mini_cluster.cc:1428] Setting key 54ea33d504cc6ea2a1c4023559d65740
Sep 29 01:00:41 dist-test-slave-k0b7 krb5kdc[5569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:41.987257 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:41.965172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:47501 (local address 127.3.21.254:45219)
0929 01:00:41.965739 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:41.965754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:41.966641 (+ 887us) server_negotiation.cc:407] Connection header received
0929 01:00:41.969507 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:41.969512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:41.969624 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:41.969745 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:41.971703 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.972519 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.974167 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.974652 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.974790 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:41.979736 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:41.979742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:41.979745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:41.979781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:41.983714 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:41.984621 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:41.984625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:41.984627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:41.984755 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:41.985278 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:41.985282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:41.985284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:41.985581 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:41.985724 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:41.986644 (+ 920us) server_negotiation.cc:299] Negotiation successful
0929 01:00:41.986832 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":249,"threads_started":1}
I20260929 01:00:41.988303 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:41.965651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45219 (local address 127.3.21.195:47501)
0929 01:00:41.966412 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:41.966449 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:41.969025 (+ 2576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:41.970055 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:41.970070 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:41.970785 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:41.971517 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:41.971537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.972690 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.972701 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:41.973995 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:41.974008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:41.974897 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:41.974905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:41.975155 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:41.976090 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:41.976119 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:41.979406 (+ 3287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:41.983894 (+ 4488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:41.983904 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:41.983920 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:41.984439 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:41.984886 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:41.984890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:41.984893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:41.985126 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:41.985733 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:41.985740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:41.986124 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 01:00:41.986396 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 01:00:41.987035 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":347,"threads_started":1}
I20260929 01:00:41.990411 6071 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45219
I20260929 01:00:41.990826 6071 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:41.991793 6071 heartbeater.cc:511] Master 127.3.21.254:45219 requested a full tablet report, sending...
I20260929 01:00:41.993686 5614 ts_manager.cc:194] Registered new tserver with Master: db2044fe320945968f02718b2a3689cd (127.3.21.195:33757)
I20260929 01:00:41.994719 5614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:47501
I20260929 01:00:41.997519 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:00:42.031437 6073 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:00:42.028460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.3.21.254:45219)
0929 01:00:42.028788 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:42.028798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:42.029243 (+ 445us) server_negotiation.cc:407] Connection header received
0929 01:00:42.029411 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:42.029415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:42.029483 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:42.029573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:42.031171 (+ 1598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57632: BlockingRecv error: recv got EOF from 127.0.0.1:57632 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 01:00:42.035862 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5675
W20260929 01:00:42.038992 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5675: Not authorized: Client connection negotiation failed: client connection to 127.3.21.193:43329: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:00:42.039179 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 5675
I20260929 01:00:42.055416 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5809
W20260929 01:00:42.058434 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5809: Not authorized: Client connection negotiation failed: client connection to 127.3.21.194:41065: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:00:42.058658 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 5809
I20260929 01:00:42.074369 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5943
W20260929 01:00:42.077908 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5943: Not authorized: Client connection negotiation failed: client connection to 127.3.21.195:33757: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:00:42.078058 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 5943
I20260929 01:00:42.094386 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 5582
I20260929 01:00:42.095901 5644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:42.205062 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 5582
2026-09-29T01:00:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3984 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:42 dist-test-slave-k0b7 krb5kdc[6089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:42 dist-test-slave-k0b7 krb5kdc[6089](info): set up 2 sockets
Sep 29 01:00:42 dist-test-slave-k0b7 krb5kdc[6089](info): commencing operation
krb5kdc: starting...
W20260929 01:00:43.342971 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:43 dist-test-slave-k0b7 krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:43.563706 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36863
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:43.908970 6102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:43.909404 6102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:43.909642 6102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:43.917755 6102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:43.917973 6102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:43.918092 6102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:43.918197 6102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:43.918282 6102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:43.930234 6102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:43.931756 6102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:43.933162 6102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:43.947654 6108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.947867 6110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:43.948464 6107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:43.948623 6102 server_base.cc:1034] running on GCE node
I20260929 01:00:43.949612 6102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:43.951454 6102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:43.952873 6102 hybrid_clock.cc:648] HybridClock initialized: now 1790643643952751 us; error 34 us; skew 500 ppm
I20260929 01:00:43.953302 6102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:43 dist-test-slave-k0b7 krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:43.957280 6102 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:43.959327 6102 webserver.cc:460] Webserver started at http://127.3.21.254:46311/ using document root <none> and password file <none>
I20260929 01:00:43.960110 6102 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:43.960273 6102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:43.960616 6102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:43.963918 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "09f1a60b42714fc593e38d3be55817eb"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-k0b7"
server_key: "8614d4aebb5d9c78e140deedb66b2af4"
server_key_iv: "0295ddd0858c8a4e4359b106991acdb7"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.964829 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "09f1a60b42714fc593e38d3be55817eb"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-k0b7"
server_key: "8614d4aebb5d9c78e140deedb66b2af4"
server_key_iv: "0295ddd0858c8a4e4359b106991acdb7"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.970991 6102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:00:43.975255 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:43.977233 6102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:00:43.977555 6102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "09f1a60b42714fc593e38d3be55817eb"
format_stamp: "Formatted at 2026-09-29 01:00:43 on dist-test-slave-k0b7"
server_key: "8614d4aebb5d9c78e140deedb66b2af4"
server_key_iv: "0295ddd0858c8a4e4359b106991acdb7"
server_key_version: "encryptionkey@0"
I20260929 01:00:43.977806 6102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:44.014088 6102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:44.019438 6102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:44.019812 6102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:44.040031 6102 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36863
I20260929 01:00:44.040131 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36863 every 8 connection(s)
I20260929 01:00:44.041525 6102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:44.045617 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6102
I20260929 01:00:44.045845 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:44.046080 6169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:44.046713 3159 external_mini_cluster.cc:1428] Setting key ac3efe849177b652cb6af4c79c4100de
I20260929 01:00:44.062454 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb: Bootstrap starting.
Sep 29 01:00:44 dist-test-slave-k0b7 krb5kdc[6089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:44.066545 6169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:44.067997 6169 log.cc:824] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:44.071358 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb: No bootstrap required, opened a new log
I20260929 01:00:44.072293 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.048791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.3.21.254:36863)
0929 01:00:44.049920 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.049933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.049983 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:00:44.051648 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.051673 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.052124 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.052513 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.054659 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.056069 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.058107 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.058977 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.059130 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.064088 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.064097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.064117 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:44.064182 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.067841 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:44.068682 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:44.068698 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:44.068705 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:44.068894 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:44.069297 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:44.069303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:44.069306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:44.069868 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.070156 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.070688 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.071142 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":533,"threads_started":1}
I20260929 01:00:44.076759 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:00:44.079856 6169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f1a60b42714fc593e38d3be55817eb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36863 } }
I20260929 01:00:44.080322 6169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:44.080435 6169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f1a60b42714fc593e38d3be55817eb, State: Initialized, Role: FOLLOWER
I20260929 01:00:44.081041 6169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f1a60b42714fc593e38d3be55817eb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36863 } }
I20260929 01:00:44.081305 6169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:44.081473 6169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:44.081655 6169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:44.085382 6169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f1a60b42714fc593e38d3be55817eb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36863 } }
I20260929 01:00:44.086187 6169 leader_election.cc:302] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f1a60b42714fc593e38d3be55817eb; no voters:
I20260929 01:00:44.087143 6169 leader_election.cc:288] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:44.087320 6175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:44.088657 6175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [term 1 LEADER]: Becoming Leader. State: Replica: 09f1a60b42714fc593e38d3be55817eb, State: Running, Role: LEADER
I20260929 01:00:44.089270 6175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f1a60b42714fc593e38d3be55817eb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36863 } }
I20260929 01:00:44.089658 6169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:44.092067 6177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f1a60b42714fc593e38d3be55817eb. Latest consensus state: current_term: 1 leader_uuid: "09f1a60b42714fc593e38d3be55817eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f1a60b42714fc593e38d3be55817eb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36863 } } }
I20260929 01:00:44.092465 6177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:44.093043 6176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f1a60b42714fc593e38d3be55817eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f1a60b42714fc593e38d3be55817eb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36863 } } }
I20260929 01:00:44.093343 6176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:44.104585 6180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:44.110643 6180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:44.129819 6180 catalog_manager.cc:1347] Generated new cluster ID: b8945a87019e41a08c8916ecfd3bab2f
I20260929 01:00:44.129966 6180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:44.140084 6196 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:00:44.150501 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:44.142379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.3.21.254:36863)
0929 01:00:44.142649 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:44.142658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:44.143269 (+ 611us) server_negotiation.cc:407] Connection header received
0929 01:00:44.143486 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:44.143492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:44.143566 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:44.143678 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:44.144739 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.145481 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.146911 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:44.147369 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:44.147520 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:44.148721 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:44.148726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:44.148730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:44.149562 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:44.149618 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:44.149654 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:44.149789 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:44.149918 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 01:00:44.150025 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260929 01:00:44.150595 6180 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:44.152465 6180 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:44.167263 6180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f1a60b42714fc593e38d3be55817eb: Generated new TSK 0
I20260929 01:00:44.168443 6180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:44.177850 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6102
I20260929 01:00:44.179093 6164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:44.274063 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6102
2026-09-29T01:00:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2064 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:44 dist-test-slave-k0b7 krb5kdc[6203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:44 dist-test-slave-k0b7 krb5kdc[6203](info): set up 2 sockets
Sep 29 01:00:44 dist-test-slave-k0b7 krb5kdc[6203](info): commencing operation
krb5kdc: starting...
W20260929 01:00:45.427623 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:45 dist-test-slave-k0b7 krb5kdc[6203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:45.647804 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:45021
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:45.993930 6216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:45.994314 6216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:45.994534 6216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:46.002416 6216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:46.002691 6216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:46.002759 6216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:46.002817 6216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:46.002861 6216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:46.013329 6216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:45021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:46.014693 6216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:46.015938 6216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:46.030615 6222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:46.030645 6221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:46.030826 6216 server_base.cc:1034] running on GCE node
W20260929 01:00:46.030655 6224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:46.032179 6216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:46.033731 6216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:46.035040 6216 hybrid_clock.cc:648] HybridClock initialized: now 1790643646035018 us; error 66 us; skew 500 ppm
I20260929 01:00:46.035444 6216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:46 dist-test-slave-k0b7 krb5kdc[6203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643646, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:46.046188 6216 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:46.048004 6216 webserver.cc:460] Webserver started at http://127.3.21.254:39359/ using document root <none> and password file <none>
I20260929 01:00:46.048718 6216 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:46.048825 6216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:46.049091 6216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:46.052268 6216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "3eb600bf25374a91a153f6ffffe583e0"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-k0b7"
server_key: "9df76d34b085ee7f50bd04ae187b6f8c"
server_key_iv: "3c2f4679c315b77baedd75f527209d9c"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.053119 6216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "3eb600bf25374a91a153f6ffffe583e0"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-k0b7"
server_key: "9df76d34b085ee7f50bd04ae187b6f8c"
server_key_iv: "3c2f4679c315b77baedd75f527209d9c"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.061151 6216 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260929 01:00:46.065248 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:46.067137 6216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:00:46.067355 6216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "3eb600bf25374a91a153f6ffffe583e0"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-k0b7"
server_key: "9df76d34b085ee7f50bd04ae187b6f8c"
server_key_iv: "3c2f4679c315b77baedd75f527209d9c"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.067590 6216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:46.082617 6216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:46.087674 6216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:46.087994 6216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:46.108906 6216 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:45021
I20260929 01:00:46.108982 6282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:45021 every 8 connection(s)
I20260929 01:00:46.110431 6216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:46.114373 6283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:46.120031 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6216
I20260929 01:00:46.120277 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:46.121075 3159 external_mini_cluster.cc:1428] Setting key b7dd471e9aafc4557a972e84325145a6
I20260929 01:00:46.126843 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0: Bootstrap starting.
I20260929 01:00:46.130851 6283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:46.132192 6283 log.cc:824] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:46.136603 6283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0: No bootstrap required, opened a new log
Sep 29 01:00:46 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:46.145058 6283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb600bf25374a91a153f6ffffe583e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45021 } }
I20260929 01:00:46.145550 6283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:46.145699 6283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb600bf25374a91a153f6ffffe583e0, State: Initialized, Role: FOLLOWER
I20260929 01:00:46.146376 6283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb600bf25374a91a153f6ffffe583e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45021 } }
I20260929 01:00:46.146680 6283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:46.146831 6283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:46.147028 6283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:46.147158 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:46.123508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.3.21.254:45021)
0929 01:00:46.124389 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:46.124401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:46.124443 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:00:46.126346 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:46.126372 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:46.126976 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:46.127384 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:46.129273 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.130828 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.132724 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.134061 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.134214 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:46.139250 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:46.139258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:46.139278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:46.139325 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:46.142587 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:46.143379 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:46.143388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:46.143407 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:46.143592 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:46.143986 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:46.143993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:46.144008 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:46.144594 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:46.144921 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:46.145358 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 01:00:46.145886 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":413,"threads_started":1}
I20260929 01:00:46.149488 6283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb600bf25374a91a153f6ffffe583e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45021 } }
I20260929 01:00:46.150034 6283 leader_election.cc:302] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eb600bf25374a91a153f6ffffe583e0; no voters:
I20260929 01:00:46.150664 6283 leader_election.cc:288] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:46.150787 6288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:46.151566 6288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [term 1 LEADER]: Becoming Leader. State: Replica: 3eb600bf25374a91a153f6ffffe583e0, State: Running, Role: LEADER
I20260929 01:00:46.152149 6288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb600bf25374a91a153f6ffffe583e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45021 } }
I20260929 01:00:46.152894 6283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:46.155028 6290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb600bf25374a91a153f6ffffe583e0. Latest consensus state: current_term: 1 leader_uuid: "3eb600bf25374a91a153f6ffffe583e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb600bf25374a91a153f6ffffe583e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45021 } } }
I20260929 01:00:46.155450 6290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:46.155249 6289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eb600bf25374a91a153f6ffffe583e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eb600bf25374a91a153f6ffffe583e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 45021 } } }
I20260929 01:00:46.155577 6289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:46.156332 6293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:46.162518 6293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:46.173717 6293 catalog_manager.cc:1347] Generated new cluster ID: 006e2eade67a4332bdba3b95292298b3
I20260929 01:00:46.173862 6293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:46.228235 6293 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:46.229650 6293 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:46.238121 6293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eb600bf25374a91a153f6ffffe583e0: Generated new TSK 0
I20260929 01:00:46.239140 6293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:46.308331 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45021
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:46.687615 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:46.687945 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:46.688054 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:46.695974 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:46.696141 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:46.696260 6309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:46.706322 6309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:46.707674 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:46.709002 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:46.724527 6315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:46.724617 6317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:46.724581 6314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:46.725332 6309 server_base.cc:1034] running on GCE node
I20260929 01:00:46.725950 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:46.727001 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:46.728253 6309 hybrid_clock.cc:648] HybridClock initialized: now 1790643646728200 us; error 45 us; skew 500 ppm
I20260929 01:00:46.728641 6309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:46 dist-test-slave-k0b7 krb5kdc[6203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643646, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:46.732359 6309 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:46.734143 6309 webserver.cc:460] Webserver started at http://127.3.21.193:33757/ using document root <none> and password file <none>
I20260929 01:00:46.734990 6309 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:46.735147 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:46.735477 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:46.738858 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "0966c057bcae4678ae8bcd7d36c8d757"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-k0b7"
server_key: "277816277256c46c481a22d99558b533"
server_key_iv: "2cddb4b0359079baa93ab0355487a494"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.739686 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "0966c057bcae4678ae8bcd7d36c8d757"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-k0b7"
server_key: "277816277256c46c481a22d99558b533"
server_key_iv: "2cddb4b0359079baa93ab0355487a494"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.745647 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:00:46.749711 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:46.751243 6309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:00:46.751454 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "0966c057bcae4678ae8bcd7d36c8d757"
format_stamp: "Formatted at 2026-09-29 01:00:46 on dist-test-slave-k0b7"
server_key: "277816277256c46c481a22d99558b533"
server_key_iv: "2cddb4b0359079baa93ab0355487a494"
server_key_version: "encryptionkey@0"
I20260929 01:00:46.751682 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:46.765820 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:46.772225 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:46.772648 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:46.773922 6309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:46.775566 6309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:46.775686 6309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:46.775869 6309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:46.775949 6309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:46.817736 6309 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:45091
I20260929 01:00:46.817801 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:45091 every 8 connection(s)
I20260929 01:00:46.819295 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:46.825452 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6309
I20260929 01:00:46.826021 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:46.827064 3159 external_mini_cluster.cc:1428] Setting key 0d523c0d587cee46623008f3bf729f19
Sep 29 01:00:46 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643646, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:46.847091 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:46.822326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:53703 (local address 127.3.21.254:45021)
0929 01:00:46.822879 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:46.822885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:46.824248 (+ 1363us) server_negotiation.cc:407] Connection header received
0929 01:00:46.826821 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:46.826827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:46.826939 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:46.827065 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:46.828951 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.829938 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.832144 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.832924 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.833092 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:46.838254 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:46.838265 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:46.838273 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:46.838327 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:46.842392 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:46.843680 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:46.843690 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:46.843696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:46.843893 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:46.844519 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:46.844527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:46.844534 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:46.844974 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:46.845172 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:46.846193 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0929 01:00:46.846464 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":337,"threads_started":1}
I20260929 01:00:46.847901 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:46.822938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45021 (local address 127.3.21.193:53703)
0929 01:00:46.823780 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:46.823980 (+ 200us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:46.826396 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:46.827365 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:46.827376 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:46.828014 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:46.828752 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:46.828772 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.830172 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.830177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:46.831926 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:46.831950 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:46.833395 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:46.833406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:46.833683 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:46.834639 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:46.834677 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:46.837871 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:46.842689 (+ 4818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:46.842701 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:46.842725 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:46.843488 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:46.844085 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:46.844092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:46.844096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:46.844361 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:46.845159 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:46.845167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:46.845624 (+ 457us) client_negotiation.cc:769] Sending connection context
0929 01:00:46.846007 (+ 383us) client_negotiation.cc:240] Negotiation successful
0929 01:00:46.846555 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":340,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:00:46.850157 6437 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45021
I20260929 01:00:46.850540 6437 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:46.851589 6437 heartbeater.cc:511] Master 127.3.21.254:45021 requested a full tablet report, sending...
I20260929 01:00:46.854250 6248 ts_manager.cc:194] Registered new tserver with Master: 0966c057bcae4678ae8bcd7d36c8d757 (127.3.21.193:45091)
I20260929 01:00:46.856038 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:53703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:46.887485 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45021
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:47.244306 6443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:47.244637 6443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:47.244807 6443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:47.252907 6443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:47.253048 6443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:47.253232 6443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:47.263955 6443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:47.265414 6443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:47.267019 6443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:47.283080 6451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.283046 6448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.283128 6449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:47.284265 6443 server_base.cc:1034] running on GCE node
I20260929 01:00:47.285112 6443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:47.286231 6443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:47.287503 6443 hybrid_clock.cc:648] HybridClock initialized: now 1790643647287441 us; error 61 us; skew 500 ppm
I20260929 01:00:47.287925 6443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:47 dist-test-slave-k0b7 krb5kdc[6203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:47.291664 6443 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:47.293684 6443 webserver.cc:460] Webserver started at http://127.3.21.194:42229/ using document root <none> and password file <none>
I20260929 01:00:47.294435 6443 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:47.294628 6443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:47.294929 6443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:47.298369 6443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "0768c350d85043eeae840632c8e024e6"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-k0b7"
server_key: "bf6bb0c193ef501ccbe47b0a68f4cfef"
server_key_iv: "f284fa87f588863568f4536ef8b88555"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.299599 6443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "0768c350d85043eeae840632c8e024e6"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-k0b7"
server_key: "bf6bb0c193ef501ccbe47b0a68f4cfef"
server_key_iv: "f284fa87f588863568f4536ef8b88555"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.308343 6443 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 01:00:47.314860 6458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.317425 6443 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:00:47.317696 6443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0768c350d85043eeae840632c8e024e6"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-k0b7"
server_key: "bf6bb0c193ef501ccbe47b0a68f4cfef"
server_key_iv: "f284fa87f588863568f4536ef8b88555"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.317962 6443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:47.339164 6443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:47.344359 6443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:47.344704 6443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:47.345968 6443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:47.348260 6443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:47.348351 6443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.348491 6443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:47.348558 6443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.388868 6443 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:44535
I20260929 01:00:47.388919 6570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:44535 every 8 connection(s)
I20260929 01:00:47.390242 6443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:47.390800 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6443
I20260929 01:00:47.391004 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:47.391636 3159 external_mini_cluster.cc:1428] Setting key 95419aebb9c57a36e1ce512042dee5c5
Sep 29 01:00:47 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:00:47.416170 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.393421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:36019 (local address 127.3.21.254:45021)
0929 01:00:47.393995 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.394001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.395192 (+ 1191us) server_negotiation.cc:407] Connection header received
0929 01:00:47.397262 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.397268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.397372 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.397497 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:47.399449 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.400423 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.403494 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.404058 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.404192 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.409262 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:47.409268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:47.409271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:47.409317 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:47.412184 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.413139 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.413151 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.413154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.413279 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.413930 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.413935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.413937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.414241 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:47.414400 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.415519 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.415713 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":200,"threads_started":1}
I20260929 01:00:47.417230 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.394116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45021 (local address 127.3.21.194:36019)
0929 01:00:47.395026 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.395068 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.396916 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.397776 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.397785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.398346 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:47.399121 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.399152 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.400619 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.400628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.403236 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.403262 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.404357 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.404367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.404728 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.405678 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:47.405722 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:47.408950 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:47.412389 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.412400 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.412423 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.412985 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.413410 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.413415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.413419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.413632 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.414419 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:47.414428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:47.414996 (+ 568us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.415339 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.415791 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":418,"threads_started":1}
I20260929 01:00:47.419075 6571 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45021
I20260929 01:00:47.419411 6571 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:47.420240 6571 heartbeater.cc:511] Master 127.3.21.254:45021 requested a full tablet report, sending...
I20260929 01:00:47.421979 6248 ts_manager.cc:194] Registered new tserver with Master: 0768c350d85043eeae840632c8e024e6 (127.3.21.194:44535)
I20260929 01:00:47.423475 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:36019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:47.443526 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45021
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:47.797787 6577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:47.798126 6577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:47.798237 6577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:47.806142 6577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:47.806306 6577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:47.806435 6577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:47.819468 6577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:45021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:47.820947 6577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:47.822569 6577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:47.839054 6582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.839128 6583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:47.839121 6585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:47.840157 6577 server_base.cc:1034] running on GCE node
I20260929 01:00:47.841166 6577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:47.842208 6577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:47.843495 6577 hybrid_clock.cc:648] HybridClock initialized: now 1790643647843377 us; error 105 us; skew 500 ppm
I20260929 01:00:47.844022 6577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:47 dist-test-slave-k0b7 krb5kdc[6203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:47.847936 6577 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:47.849738 6577 webserver.cc:460] Webserver started at http://127.3.21.195:45561/ using document root <none> and password file <none>
I20260929 01:00:47.850579 6577 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:47.850732 6577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:47.851060 6577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:47.854338 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "d85e5acc613a49948eb66aee9c4ca15e"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-k0b7"
server_key: "f075ddb48e8b49384e9a09b6a6f522bb"
server_key_iv: "e26bcd3c47a138ca11cda5b288042872"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.855229 6577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "d85e5acc613a49948eb66aee9c4ca15e"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-k0b7"
server_key: "f075ddb48e8b49384e9a09b6a6f522bb"
server_key_iv: "e26bcd3c47a138ca11cda5b288042872"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.860875 6437 heartbeater.cc:503] Master 127.3.21.254:45021 was elected leader, sending a full tablet report...
I20260929 01:00:47.861093 6577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:00:47.865316 6592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.867372 6577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:47.867630 6577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "d85e5acc613a49948eb66aee9c4ca15e"
format_stamp: "Formatted at 2026-09-29 01:00:47 on dist-test-slave-k0b7"
server_key: "f075ddb48e8b49384e9a09b6a6f522bb"
server_key_iv: "e26bcd3c47a138ca11cda5b288042872"
server_key_version: "encryptionkey@0"
I20260929 01:00:47.868083 6577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:47.909140 6577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:47.914603 6577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:47.915014 6577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:47.916208 6577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:47.917891 6577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:47.918012 6577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.918195 6577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:47.918277 6577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:47.957404 6577 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:46819
I20260929 01:00:47.957500 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:46819 every 8 connection(s)
I20260929 01:00:47.958901 6577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:47.966811 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6577
I20260929 01:00:47.967000 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:47.967478 3159 external_mini_cluster.cc:1428] Setting key da5ff79ea4a1631264b0239c8cdf0891
Sep 29 01:00:47 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:47.986760 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.962385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:45021 (local address 127.3.21.195:59907)
0929 01:00:47.963350 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:47.963405 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:47.965260 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:47.966154 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:47.966164 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:47.966852 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:47.967701 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.967723 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.969225 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.969232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.971324 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:47.971339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.972452 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.972460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:47.972721 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.973636 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:47.973662 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:47.976347 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:47.980146 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.980159 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.980184 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.981414 (+ 1230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.981929 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:47.981934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:47.981938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:47.982249 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:47.983115 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:47.983125 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:47.983674 (+ 549us) client_negotiation.cc:769] Sending connection context
0929 01:00:47.984020 (+ 346us) client_negotiation.cc:240] Negotiation successful
0929 01:00:47.984790 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":433,"threads_started":1}
I20260929 01:00:47.986788 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:47.961902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:59907 (local address 127.3.21.254:45021)
0929 01:00:47.962558 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:47.962572 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:47.963561 (+ 989us) server_negotiation.cc:407] Connection header received
0929 01:00:47.965604 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:47.965611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:47.965719 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:47.965853 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:47.967913 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.968975 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.971504 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:47.972134 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:47.972289 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:47.976608 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:47.976613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:47.976617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:47.976654 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:47.979787 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.981606 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.981612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.981616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.981790 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:47.982433 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:47.982442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:47.982448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:47.982968 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:47.983138 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:47.985968 (+ 2830us) server_negotiation.cc:299] Negotiation successful
0929 01:00:47.986252 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":346,"threads_started":1}
I20260929 01:00:47.988617 6705 heartbeater.cc:347] Connected to a master server at 127.3.21.254:45021
I20260929 01:00:47.989108 6705 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:47.990298 6705 heartbeater.cc:511] Master 127.3.21.254:45021 requested a full tablet report, sending...
I20260929 01:00:47.992738 6248 ts_manager.cc:194] Registered new tserver with Master: d85e5acc613a49948eb66aee9c4ca15e (127.3.21.195:46819)
I20260929 01:00:47.993778 6248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:59907
I20260929 01:00:47.997570 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643648, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643648, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:48.063877 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.047289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.3.21.254:45021)
0929 01:00:48.047506 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.047511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.048096 (+ 585us) server_negotiation.cc:407] Connection header received
0929 01:00:48.048330 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.048334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.048434 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.048542 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.050086 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.050998 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.052236 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.052637 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.052772 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.057773 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.057780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.057785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:48.057829 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.061237 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.061958 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.061963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.061965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.062080 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.062556 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.062560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.062562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.062926 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.063104 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.063464 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.063637 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 01:00:48.064519 6248 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58430
I20260929 01:00:48.066287 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6309
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643648, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:48.085363 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.067538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.3.21.193:45091)
0929 01:00:48.068324 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.068331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.068356 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:00:48.068412 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.068418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.068624 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.068746 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.070145 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.071559 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.073281 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.074108 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.074222 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.078699 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.078714 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.078730 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:48.078780 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.081832 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.082760 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.082768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.082771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.082952 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.083425 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.083429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.083431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.083833 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.084062 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.084626 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.085003 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":443,"threads_started":1}
W20260929 01:00:48.086880 6432 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38600
W20260929 01:00:48.088207 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6309: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:48.088425 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6309
I20260929 01:00:48.104532 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6443
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643648, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:48.123220 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.105566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.3.21.194:44535)
0929 01:00:48.106153 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.106162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.106189 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:00:48.106257 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.106263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.106471 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.106698 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.107993 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.109080 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.110651 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.111234 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.111392 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.116284 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.116300 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.116318 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:48.116365 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.119898 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.120724 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.120732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.120735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.120888 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.121296 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.121303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.121306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.121680 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.122015 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.122353 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.122684 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":343,"threads_started":1}
W20260929 01:00:48.125552 6566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41884
W20260929 01:00:48.127383 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6443: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:48.127795 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6443
I20260929 01:00:48.146806 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6577
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643648, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:48.166848 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:48.147953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.3.21.195:46819)
0929 01:00:48.148699 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:48.148706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:48.148739 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:00:48.148816 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:48.148825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:48.149059 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:48.149207 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:48.150888 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.152052 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.153575 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:48.154391 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:48.154644 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:48.160145 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:48.160163 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:48.160230 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:48.160288 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:48.163693 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.164602 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.164611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.164613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.164745 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:48.165129 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:48.165132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:48.165134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:48.165495 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:48.165730 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:48.166107 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 01:00:48.166448 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":398,"threads_started":1}
W20260929 01:00:48.168318 6700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58432
W20260929 01:00:48.169629 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:00:48.169799 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6577
I20260929 01:00:48.185622 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6216
I20260929 01:00:48.187058 6278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:48.295276 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6216
2026-09-29T01:00:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4027 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6723](info): set up 2 sockets
Sep 29 01:00:48 dist-test-slave-k0b7 krb5kdc[6723](info): commencing operation
krb5kdc: starting...
W20260929 01:00:49.443311 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:49 dist-test-slave-k0b7 krb5kdc[6723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:49.679675 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:39835
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:50.049539 6736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:50.049883 6736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:50.049976 6736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:50.057744 6736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:50.057894 6736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:50.057943 6736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:50.057993 6736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:50.058029 6736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:50.069226 6736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:50.070649 6736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:50.072134 6736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:50.086876 6741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.087198 6744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.087193 6742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:50.088305 6736 server_base.cc:1034] running on GCE node
I20260929 01:00:50.089212 6736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:50.091028 6736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:50.092262 6736 hybrid_clock.cc:648] HybridClock initialized: now 1790643650092212 us; error 45 us; skew 500 ppm
I20260929 01:00:50.092666 6736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:50 dist-test-slave-k0b7 krb5kdc[6723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:50.097000 6736 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:50.099092 6736 webserver.cc:460] Webserver started at http://127.3.21.254:44409/ using document root <none> and password file <none>
I20260929 01:00:50.099929 6736 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:50.100098 6736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:50.100443 6736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:50.103963 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "2028a2d7d1f04bf697732798ce88cc52"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-k0b7"
server_key: "4c827406ecd6ac163bfe18fe062ea571"
server_key_iv: "01f9807334975699016bd7778294bf66"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.104899 6736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "2028a2d7d1f04bf697732798ce88cc52"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-k0b7"
server_key: "4c827406ecd6ac163bfe18fe062ea571"
server_key_iv: "01f9807334975699016bd7778294bf66"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.111073 6736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:00:50.115481 6751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.117377 6736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:00:50.117640 6736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "2028a2d7d1f04bf697732798ce88cc52"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-k0b7"
server_key: "4c827406ecd6ac163bfe18fe062ea571"
server_key_iv: "01f9807334975699016bd7778294bf66"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.117900 6736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:50.143684 6736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:50.149349 6736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:50.149746 6736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:50.172839 6736 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:39835
I20260929 01:00:50.172953 6802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:39835 every 8 connection(s)
I20260929 01:00:50.174569 6736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:50.178939 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:50.183660 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6736
I20260929 01:00:50.183943 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:50.184648 3159 external_mini_cluster.cc:1428] Setting key 66a85e2cc6fc863c11d432d42c048f5b
I20260929 01:00:50.191849 6803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52: Bootstrap starting.
I20260929 01:00:50.196346 6803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:50.197652 6803 log.cc:824] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:50.201749 6803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52: No bootstrap required, opened a new log
Sep 29 01:00:50 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:50.210947 6803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028a2d7d1f04bf697732798ce88cc52" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39835 } }
I20260929 01:00:50.211474 6803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:50.211633 6803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2028a2d7d1f04bf697732798ce88cc52, State: Initialized, Role: FOLLOWER
I20260929 01:00:50.212431 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.187158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.3.21.254:39835)
0929 01:00:50.188082 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:50.188095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:50.188139 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:00:50.190155 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:50.190185 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:50.190760 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:50.191140 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:50.193201 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.194791 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.196853 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.197611 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.197769 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.203792 (+ 6023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:50.203803 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:50.203831 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:50.203885 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:50.207451 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.208347 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.208355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.208363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.208551 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.209014 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.209026 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.209030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.209674 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:50.210022 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:50.210533 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 01:00:50.210989 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":428,"threads_started":1}
I20260929 01:00:50.212337 6803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028a2d7d1f04bf697732798ce88cc52" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39835 } }
I20260929 01:00:50.212625 6803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:50.212779 6803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:50.212952 6803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:50.215822 6803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028a2d7d1f04bf697732798ce88cc52" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39835 } }
I20260929 01:00:50.216408 6803 leader_election.cc:302] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2028a2d7d1f04bf697732798ce88cc52; no voters:
I20260929 01:00:50.217113 6803 leader_election.cc:288] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:50.217226 6808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:50.218680 6808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [term 1 LEADER]: Becoming Leader. State: Replica: 2028a2d7d1f04bf697732798ce88cc52, State: Running, Role: LEADER
I20260929 01:00:50.219386 6808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028a2d7d1f04bf697732798ce88cc52" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39835 } }
I20260929 01:00:50.219658 6803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:50.222687 6809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2028a2d7d1f04bf697732798ce88cc52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028a2d7d1f04bf697732798ce88cc52" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39835 } } }
I20260929 01:00:50.223102 6809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:50.223886 6810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2028a2d7d1f04bf697732798ce88cc52. Latest consensus state: current_term: 1 leader_uuid: "2028a2d7d1f04bf697732798ce88cc52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2028a2d7d1f04bf697732798ce88cc52" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39835 } } }
I20260929 01:00:50.224145 6815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:50.224216 6810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:50.235491 6815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:50.246680 6815 catalog_manager.cc:1347] Generated new cluster ID: 32dde5b0541246afa2004a637c12550f
I20260929 01:00:50.246861 6815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:50.260773 6815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:50.262821 6815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:50.275357 6815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2028a2d7d1f04bf697732798ce88cc52: Generated new TSK 0
I20260929 01:00:50.276371 6815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:50.343643 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39835
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:50.698235 6829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:50.698616 6829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:50.698814 6829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:50.706398 6829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:50.706609 6829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:50.706817 6829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:50.717499 6829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:50.719003 6829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:50.720633 6829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:50.735374 6837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.735316 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:50.735410 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:50.736135 6829 server_base.cc:1034] running on GCE node
I20260929 01:00:50.736763 6829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:50.737730 6829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:50.738993 6829 hybrid_clock.cc:648] HybridClock initialized: now 1790643650738911 us; error 69 us; skew 500 ppm
I20260929 01:00:50.739447 6829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:50 dist-test-slave-k0b7 krb5kdc[6723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:50.743292 6829 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:50.745200 6829 webserver.cc:460] Webserver started at http://127.3.21.193:36003/ using document root <none> and password file <none>
I20260929 01:00:50.746001 6829 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:50.746158 6829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:50.746532 6829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:50.749801 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "813dc1aab13a4a68b4f02350ca5c10e3"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-k0b7"
server_key: "c7bca9c079a78b584873ba63fbeb74a9"
server_key_iv: "08c4342f3fee33067be0d0559434de86"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.750727 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "813dc1aab13a4a68b4f02350ca5c10e3"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-k0b7"
server_key: "c7bca9c079a78b584873ba63fbeb74a9"
server_key_iv: "08c4342f3fee33067be0d0559434de86"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.756443 6829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:00:50.760850 6844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.762786 6829 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:50.763173 6829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "813dc1aab13a4a68b4f02350ca5c10e3"
format_stamp: "Formatted at 2026-09-29 01:00:50 on dist-test-slave-k0b7"
server_key: "c7bca9c079a78b584873ba63fbeb74a9"
server_key_iv: "08c4342f3fee33067be0d0559434de86"
server_key_version: "encryptionkey@0"
I20260929 01:00:50.763556 6829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:50.784511 6829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:50.789331 6829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:50.789656 6829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:50.790815 6829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:50.792299 6829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:50.792392 6829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.792522 6829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:50.792594 6829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:50.839149 6829 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:43579
I20260929 01:00:50.839169 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:43579 every 8 connection(s)
I20260929 01:00:50.840519 6829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:50.842763 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6829
I20260929 01:00:50.842998 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:50.843823 3159 external_mini_cluster.cc:1428] Setting key ed9683ea538da17262599049d1c15e83
Sep 29 01:00:50 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:00:50.864171 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.843513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:60839 (local address 127.3.21.254:39835)
0929 01:00:50.843999 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:50.844007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:50.845275 (+ 1268us) server_negotiation.cc:407] Connection header received
0929 01:00:50.847346 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:50.847351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:50.847462 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:50.847581 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:50.849288 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.850010 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.851743 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.852154 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.852308 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.856401 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:50.856410 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:50.856418 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:50.856467 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:50.859917 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.861228 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.861233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.861235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.861376 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:50.861948 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:50.861951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:50.861953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:50.862234 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:50.862346 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:50.863689 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0929 01:00:50.863878 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260929 01:00:50.865101 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:50.844202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39835 (local address 127.3.21.193:60839)
0929 01:00:50.845098 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:50.845149 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:50.847035 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:50.847855 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:50.847865 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:50.848436 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:50.849085 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.849104 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.850183 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.850189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.851576 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:50.851590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:50.852395 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:50.852404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:50.852656 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:50.853408 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:50.853430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:50.856047 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:50.860128 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.860145 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.860174 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.861024 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.861513 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:50.861519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:50.861523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:50.861816 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:50.862382 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:50.862394 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:50.862971 (+ 577us) client_negotiation.cc:769] Sending connection context
0929 01:00:50.863357 (+ 386us) client_negotiation.cc:240] Negotiation successful
0929 01:00:50.863855 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:50.867107 6957 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39835
I20260929 01:00:50.867465 6957 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:50.868444 6957 heartbeater.cc:511] Master 127.3.21.254:39835 requested a full tablet report, sending...
I20260929 01:00:50.871199 6768 ts_manager.cc:194] Registered new tserver with Master: 813dc1aab13a4a68b4f02350ca5c10e3 (127.3.21.193:43579)
I20260929 01:00:50.873656 6768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:60839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:50.900347 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39835
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:51.261662 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:51.262038 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:51.262244 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:51.271765 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:51.272217 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:51.272617 6963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:51.284907 6963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:51.286701 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:51.288164 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:51.304809 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.304919 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.304917 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:51.305742 6963 server_base.cc:1034] running on GCE node
I20260929 01:00:51.306638 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:51.307713 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:51.308950 6963 hybrid_clock.cc:648] HybridClock initialized: now 1790643651308880 us; error 64 us; skew 500 ppm
I20260929 01:00:51.309334 6963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:51 dist-test-slave-k0b7 krb5kdc[6723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:51.322094 6963 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:51.324100 6963 webserver.cc:460] Webserver started at http://127.3.21.194:33639/ using document root <none> and password file <none>
I20260929 01:00:51.325141 6963 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:51.325336 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:51.325676 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:51.329034 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "8916992e268c4da99e66872e727baa33"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-k0b7"
server_key: "84e4e6c2654bb9963159ae404805a055"
server_key_iv: "36e4e0a5e81bba1c351d433adb0e9f38"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.329916 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "8916992e268c4da99e66872e727baa33"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-k0b7"
server_key: "84e4e6c2654bb9963159ae404805a055"
server_key_iv: "36e4e0a5e81bba1c351d433adb0e9f38"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.336146 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:00:51.340451 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.342092 6963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 01:00:51.342384 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "8916992e268c4da99e66872e727baa33"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-k0b7"
server_key: "84e4e6c2654bb9963159ae404805a055"
server_key_iv: "36e4e0a5e81bba1c351d433adb0e9f38"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.342653 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:51.374750 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:51.380115 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:51.380442 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:51.381476 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:51.383024 6963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:51.383103 6963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.383224 6963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:51.383298 6963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.420985 6963 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:42005
I20260929 01:00:51.421068 7090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:42005 every 8 connection(s)
I20260929 01:00:51.422585 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:51.424028 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 6963
I20260929 01:00:51.424242 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:51.425086 3159 external_mini_cluster.cc:1428] Setting key aececce84f6193bc1b73846a622f8a7f
Sep 29 01:00:51 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:00:51.453775 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.426655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:32927 (local address 127.3.21.254:39835)
0929 01:00:51.427171 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.427178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.428502 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 01:00:51.432072 (+ 3570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.432077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.432173 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.432297 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:51.434969 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.436426 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.438386 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.439262 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.439444 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.444650 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:51.444661 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:51.444667 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:51.444728 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:51.448796 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:51.449870 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:51.449876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:51.449878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:51.450088 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:51.451016 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:51.451021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:51.451023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:51.451469 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:51.451614 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:51.452732 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0929 01:00:51.453055 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
I20260929 01:00:51.454741 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.427402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39835 (local address 127.3.21.194:32927)
0929 01:00:51.428293 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.428359 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.431363 (+ 3004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.432628 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.432645 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.433579 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:51.434661 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.434699 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.436663 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.436668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.438165 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.438183 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.439554 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.439559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.440130 (+ 571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.440975 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:51.441001 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:51.444233 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:51.449021 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:51.449033 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:51.449061 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:51.449672 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:51.450367 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:51.450371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:51.450374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:51.450718 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:51.451668 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:51.451676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:51.452093 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 01:00:51.452554 (+ 461us) client_negotiation.cc:240] Negotiation successful
0929 01:00:51.453114 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":340,"threads_started":1}
I20260929 01:00:51.456893 7091 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39835
I20260929 01:00:51.457288 7091 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:51.458179 7091 heartbeater.cc:511] Master 127.3.21.254:39835 requested a full tablet report, sending...
I20260929 01:00:51.460222 6768 ts_manager.cc:194] Registered new tserver with Master: 8916992e268c4da99e66872e727baa33 (127.3.21.194:42005)
I20260929 01:00:51.461930 6768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:32927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:51.486692 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39835
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:51.841236 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:51.841562 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:51.841741 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:51.849861 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:51.850076 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:51.850256 7097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:51.860682 7097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:51.862087 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:51.863521 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:00:51.877972 6957 heartbeater.cc:503] Master 127.3.21.254:39835 was elected leader, sending a full tablet report...
W20260929 01:00:51.878193 7103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.878427 7105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:51.878958 7102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:51.880184 7097 server_base.cc:1034] running on GCE node
I20260929 01:00:51.881052 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:51.882233 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:51.883463 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790643651883417 us; error 39 us; skew 500 ppm
I20260929 01:00:51.883880 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:51 dist-test-slave-k0b7 krb5kdc[6723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:51.887892 7097 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:51.890182 7097 webserver.cc:460] Webserver started at http://127.3.21.195:36955/ using document root <none> and password file <none>
I20260929 01:00:51.891114 7097 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:51.891273 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:51.891597 7097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:51.896107 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "ac7522997eac4dd5aaed61c377377afa"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-k0b7"
server_key: "9276db46878457955706cee1b2518bf5"
server_key_iv: "62227187c3eaff94d8f7bbf7d51b4d93"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.897053 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "ac7522997eac4dd5aaed61c377377afa"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-k0b7"
server_key: "9276db46878457955706cee1b2518bf5"
server_key_iv: "62227187c3eaff94d8f7bbf7d51b4d93"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.903692 7097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:00:51.908512 7112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.910636 7097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:00:51.910881 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "ac7522997eac4dd5aaed61c377377afa"
format_stamp: "Formatted at 2026-09-29 01:00:51 on dist-test-slave-k0b7"
server_key: "9276db46878457955706cee1b2518bf5"
server_key_iv: "62227187c3eaff94d8f7bbf7d51b4d93"
server_key_version: "encryptionkey@0"
I20260929 01:00:51.911160 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:51.929379 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:51.934588 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:51.934978 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:51.936215 7097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:51.937774 7097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:51.937889 7097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.938040 7097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:51.938107 7097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:51.975055 7097 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:39237
I20260929 01:00:51.975157 7224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:39237 every 8 connection(s)
I20260929 01:00:51.976449 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:51.979270 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7097
I20260929 01:00:51.979521 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:51.980121 3159 external_mini_cluster.cc:1428] Setting key b85cf16cadae7dbf7d2ce4cb987ba1df
Sep 29 01:00:51 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:52.001113 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.980414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:41055 (local address 127.3.21.254:39835)
0929 01:00:51.981378 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:51.981384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:51.982222 (+ 838us) server_negotiation.cc:407] Connection header received
0929 01:00:51.984386 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:51.984390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:51.984469 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:51.984570 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:51.986345 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.987420 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.989266 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.989793 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.989913 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.993719 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:51.993726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:51.993730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:51.993774 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:51.997208 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:51.998228 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:51.998232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:51.998235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:51.998380 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:51.999088 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:51.999092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:51.999094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:51.999441 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:51.999583 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.000492 (+ 909us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.000692 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":609,"threads_started":1}
I20260929 01:00:52.002462 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:51.981052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39835 (local address 127.3.21.195:41055)
0929 01:00:51.982054 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:51.982114 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:51.984072 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:51.984896 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:51.984906 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:51.985545 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:51.986144 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.986163 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.987614 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.987618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.989079 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:51.989093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:51.990014 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:51.990019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:51.990231 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:51.991064 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:51.991088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:51.993441 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:51.997390 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:51.997399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:51.997417 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:51.998047 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:51.998721 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:51.998725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:51.998728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:51.998938 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:51.999636 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:51.999643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:52.000014 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 01:00:52.000313 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 01:00:52.000966 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":418,"threads_started":1}
I20260929 01:00:52.004941 7225 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39835
I20260929 01:00:52.005281 7225 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:52.006110 7225 heartbeater.cc:511] Master 127.3.21.254:39835 requested a full tablet report, sending...
I20260929 01:00:52.008764 6768 ts_manager.cc:194] Registered new tserver with Master: ac7522997eac4dd5aaed61c377377afa (127.3.21.195:39237)
I20260929 01:00:52.009838 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:52.009824 6768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:41055
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[6723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:52.064706 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.047451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.3.21.193:43579)
0929 01:00:52.048138 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.048146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.048242 (+ 96us) server_negotiation.cc:407] Connection header received
0929 01:00:52.048387 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.048395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.048619 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.048753 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.050273 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.051439 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.052892 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.053515 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.053624 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.057983 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.057999 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.058016 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:52.058064 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.061355 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.062324 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.062333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.062336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.062475 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.063030 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.063034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.063036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.063472 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.063706 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.064014 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.064219 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":320,"threads_started":1}
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:52.087219 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.070454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.3.21.254:39835)
0929 01:00:52.070770 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.070780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.071412 (+ 632us) server_negotiation.cc:407] Connection header received
0929 01:00:52.071638 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.071646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.071727 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.071843 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.073157 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.073913 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.075422 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.075886 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.076027 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.080476 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.080481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.080485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:52.080528 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.083917 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.084872 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.084876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.084878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.085143 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.085584 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.085588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.085590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.085980 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.086173 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.086754 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.086960 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260929 01:00:52.088022 6768 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750
I20260929 01:00:52.089217 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6829
I20260929 01:00:52.105003 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.090214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.3.21.193:43579)
0929 01:00:52.090454 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.090461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.090813 (+ 352us) server_negotiation.cc:407] Connection header received
0929 01:00:52.091106 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.091110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.091224 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.091313 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.092699 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.093435 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.095016 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.095522 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.095689 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.098964 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.098970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.098974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:52.099024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.101997 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.102975 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.102980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.102982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.103128 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.103539 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.103543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.103545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.103852 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.104117 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.104436 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.104679 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:00:52.105646 6952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:52.194979 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6829
I20260929 01:00:52.214633 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6963
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:52.233585 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.215807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.3.21.194:42005)
0929 01:00:52.216382 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.216390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.216417 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:00:52.216484 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.216490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.216913 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.217113 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.218878 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.220023 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.221367 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.221952 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.222096 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.227081 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.227098 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.227115 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:52.227182 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.230402 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.231289 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.231298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.231300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.231461 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.231885 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.231890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.231891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.232261 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.232579 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.232919 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.233202 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":302,"threads_started":1}
I20260929 01:00:52.235057 7086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:52.326408 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6963
I20260929 01:00:52.343613 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7097
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[6723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:52.363243 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:52.344987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.3.21.195:39237)
0929 01:00:52.345745 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:52.345753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:52.345786 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:00:52.345853 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:52.345860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:52.346092 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:52.346258 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:52.347702 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.348819 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.350192 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:52.350892 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:52.351005 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:52.356578 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:52.356593 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:52.356610 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:52.356652 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:52.359890 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.360777 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.360788 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.360790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.360939 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:52.361398 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:52.361402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:52.361404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:52.361899 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:52.362143 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:52.362415 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 01:00:52.362812 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":350,"threads_started":1}
I20260929 01:00:52.364782 7220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:52.453214 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7097
I20260929 01:00:52.469619 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 6736
I20260929 01:00:52.471715 6798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:52.580190 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 6736
2026-09-29T01:00:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4286 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[7245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[7245](info): set up 2 sockets
Sep 29 01:00:52 dist-test-slave-k0b7 krb5kdc[7245](info): commencing operation
krb5kdc: starting...
W20260929 01:00:53.735322 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:00:53 dist-test-slave-k0b7 krb5kdc[7245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:00:53.969219 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:33369
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:54.318323 7258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:54.318760 7258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:54.318974 7258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:54.327675 7258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:54.327819 7258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:54.327937 7258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:54.328100 7258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:54.328184 7258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:54.339999 7258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:33369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:54.341500 7258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:54.343029 7258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:54.357020 7263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:54.356954 7264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:54.356995 7266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:54.358367 7258 server_base.cc:1034] running on GCE node
I20260929 01:00:54.359382 7258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:54.360992 7258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:54.362258 7258 hybrid_clock.cc:648] HybridClock initialized: now 1790643654362179 us; error 57 us; skew 500 ppm
I20260929 01:00:54.362807 7258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:54 dist-test-slave-k0b7 krb5kdc[7245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:54.366997 7258 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:00:54.368477 7258 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:54.368637 7258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:54.369011 7258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:54.372587 7258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "1166c2a0e5e34cfaa22437825b1da122"
format_stamp: "Formatted at 2026-09-29 01:00:54 on dist-test-slave-k0b7"
server_key: "099d9deacf24148b4e0f39aad5082a1d"
server_key_iv: "0b8c3cc16a803e407826dd94c609320e"
server_key_version: "encryptionkey@0"
I20260929 01:00:54.373600 7258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "1166c2a0e5e34cfaa22437825b1da122"
format_stamp: "Formatted at 2026-09-29 01:00:54 on dist-test-slave-k0b7"
server_key: "099d9deacf24148b4e0f39aad5082a1d"
server_key_iv: "0b8c3cc16a803e407826dd94c609320e"
server_key_version: "encryptionkey@0"
I20260929 01:00:54.379863 7258 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:00:54.384428 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:54.386365 7258 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:00:54.386617 7258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "1166c2a0e5e34cfaa22437825b1da122"
format_stamp: "Formatted at 2026-09-29 01:00:54 on dist-test-slave-k0b7"
server_key: "099d9deacf24148b4e0f39aad5082a1d"
server_key_iv: "0b8c3cc16a803e407826dd94c609320e"
server_key_version: "encryptionkey@0"
I20260929 01:00:54.386893 7258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:54.403859 7258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:54.408756 7258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:54.409217 7258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:54.431845 7258 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:33369
I20260929 01:00:54.431888 7323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:33369 every 8 connection(s)
I20260929 01:00:54.432989 7258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:54.437223 7324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:54.441955 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7258
I20260929 01:00:54.442219 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:54.443022 3159 external_mini_cluster.cc:1428] Setting key 23b7b7c0e50e3ea164251380ff220037
I20260929 01:00:54.450129 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122: Bootstrap starting.
I20260929 01:00:54.454242 7324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:54.455781 7324 log.cc:824] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:54.459798 7324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122: No bootstrap required, opened a new log
Sep 29 01:00:54 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:54.468225 7324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1166c2a0e5e34cfaa22437825b1da122" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33369 } }
I20260929 01:00:54.468734 7324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:54.468830 7324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1166c2a0e5e34cfaa22437825b1da122, State: Initialized, Role: FOLLOWER
I20260929 01:00:54.469538 7324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1166c2a0e5e34cfaa22437825b1da122" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33369 } }
I20260929 01:00:54.469801 7324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:54.469947 7324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:54.470057 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:54.445415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.3.21.254:33369)
0929 01:00:54.446315 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:54.446328 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:54.446376 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:00:54.448346 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:54.448375 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:54.448860 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:54.449235 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:54.451331 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:54.452875 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:54.455144 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:54.455995 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:54.456193 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:54.461566 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:54.461577 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:54.461602 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:54.461655 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:54.465319 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:54.466129 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:54.466136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:54.466148 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:54.466344 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:54.466830 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:54.466834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:54.466836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:54.467456 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:54.467773 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:54.468231 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 01:00:54.468755 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":437,"threads_started":1}
I20260929 01:00:54.470140 7324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:54.472743 7324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1166c2a0e5e34cfaa22437825b1da122" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33369 } }
I20260929 01:00:54.473330 7324 leader_election.cc:302] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1166c2a0e5e34cfaa22437825b1da122; no voters:
I20260929 01:00:54.473896 7324 leader_election.cc:288] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:54.474080 7329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:54.475026 7329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [term 1 LEADER]: Becoming Leader. State: Replica: 1166c2a0e5e34cfaa22437825b1da122, State: Running, Role: LEADER
I20260929 01:00:54.475585 7329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1166c2a0e5e34cfaa22437825b1da122" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33369 } }
I20260929 01:00:54.476076 7324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:54.478844 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1166c2a0e5e34cfaa22437825b1da122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1166c2a0e5e34cfaa22437825b1da122" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33369 } } }
I20260929 01:00:54.478906 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1166c2a0e5e34cfaa22437825b1da122. Latest consensus state: current_term: 1 leader_uuid: "1166c2a0e5e34cfaa22437825b1da122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1166c2a0e5e34cfaa22437825b1da122" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33369 } } }
I20260929 01:00:54.479228 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:54.479272 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:54.481158 7335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:54.487092 7335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:54.497450 7335 catalog_manager.cc:1347] Generated new cluster ID: 9165690dbcec4931839a463783eeea71
I20260929 01:00:54.497605 7335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:54.522469 7335 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:54.524109 7335 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:54.540810 7335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122: Generated new TSK 0
I20260929 01:00:54.541787 7335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:00:54.595708 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:33369
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:54.962153 7350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:54.962541 7350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:54.962729 7350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:54.971442 7350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:54.971575 7350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:54.971716 7350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:54.983908 7350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:33369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:54.985360 7350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:54.987228 7350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:55.004563 7356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:55.005082 7355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:55.005162 7358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:55.005504 7350 server_base.cc:1034] running on GCE node
I20260929 01:00:55.006629 7350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:55.008059 7350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:55.009364 7350 hybrid_clock.cc:648] HybridClock initialized: now 1790643655009290 us; error 50 us; skew 500 ppm
I20260929 01:00:55.009922 7350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:55 dist-test-slave-k0b7 krb5kdc[7245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:55.014830 7350 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:00:55.016640 7350 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:55.016786 7350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:55.017221 7350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:55.022351 7350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "7263b2cc2c344c08bc7f905ccde42511"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-k0b7"
server_key: "87c877bc74b22c7d9f6e497a10b139d7"
server_key_iv: "5bbf9d4a45d6ccf605e62d92fb8792f9"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.023754 7350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "7263b2cc2c344c08bc7f905ccde42511"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-k0b7"
server_key: "87c877bc74b22c7d9f6e497a10b139d7"
server_key_iv: "5bbf9d4a45d6ccf605e62d92fb8792f9"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.031112 7350 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 01:00:55.035624 7364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.037384 7350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:00:55.037637 7350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "7263b2cc2c344c08bc7f905ccde42511"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-k0b7"
server_key: "87c877bc74b22c7d9f6e497a10b139d7"
server_key_iv: "5bbf9d4a45d6ccf605e62d92fb8792f9"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.037920 7350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:55.056744 7350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:55.061884 7350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:55.062215 7350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:55.063433 7350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:55.064913 7350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:55.065037 7350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.065182 7350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:55.065291 7350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.103220 7350 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:40241
I20260929 01:00:55.103220 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:40241 every 8 connection(s)
I20260929 01:00:55.104261 7350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:55.108767 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7350
I20260929 01:00:55.109014 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:55.109797 3159 external_mini_cluster.cc:1428] Setting key ade25d965e980657b54463503a9b13fd
Sep 29 01:00:55 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:55.130312 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:55.107733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:39083 (local address 127.3.21.254:33369)
0929 01:00:55.108368 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:55.108373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:55.109492 (+ 1119us) server_negotiation.cc:407] Connection header received
0929 01:00:55.111668 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:55.111673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:55.111761 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:55.111896 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:55.113669 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.114420 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.116168 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.116611 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.116760 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:55.122311 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:55.122317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:55.122321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:55.122363 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:55.125843 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:55.127234 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:55.127239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:55.127241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:55.127382 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:55.128178 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:55.128183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:55.128185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:55.128506 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:55.128663 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:55.129808 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0929 01:00:55.130018 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":391,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:00:55.132282 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:55.108148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:33369 (local address 127.3.21.193:39083)
0929 01:00:55.109305 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:55.109356 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:55.111321 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:55.112184 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:55.112195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:55.112797 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:55.113452 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:55.113478 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.114650 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.114656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:55.115971 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:55.115986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.116869 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.116878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:55.117142 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:55.118074 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:55.118125 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:55.121958 (+ 3833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:55.126072 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:55.126089 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:55.126116 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:55.127033 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:55.127583 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:55.127593 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:55.127601 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:55.127925 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:55.128667 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:55.128680 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:55.129256 (+ 576us) client_negotiation.cc:769] Sending connection context
0929 01:00:55.129647 (+ 391us) client_negotiation.cc:240] Negotiation successful
0929 01:00:55.130329 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":493,"threads_started":1}
I20260929 01:00:55.134442 7477 heartbeater.cc:347] Connected to a master server at 127.3.21.254:33369
I20260929 01:00:55.134915 7477 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:00:55.135831 7477 heartbeater.cc:511] Master 127.3.21.254:33369 requested a full tablet report, sending...
I20260929 01:00:55.138476 7289 ts_manager.cc:194] Registered new tserver with Master: 7263b2cc2c344c08bc7f905ccde42511 (127.3.21.193:40241)
I20260929 01:00:55.140872 7289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:39083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:00:55.167470 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:33369
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:55.526754 7483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:55.527120 7483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:55.527282 7483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:55.535651 7483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:55.535830 7483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:55.536031 7483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:55.546614 7483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:33369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:55.548094 7483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:55.549425 7483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:55.563530 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:55.563489 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:55.563596 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:55.564540 7483 server_base.cc:1034] running on GCE node
I20260929 01:00:55.565444 7483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:55.566737 7483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:55.568017 7483 hybrid_clock.cc:648] HybridClock initialized: now 1790643655567972 us; error 57 us; skew 500 ppm
I20260929 01:00:55.568396 7483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:55 dist-test-slave-k0b7 krb5kdc[7245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:55.572026 7483 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:00:55.573659 7483 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:55.573815 7483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:55.574172 7483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:55.577677 7483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "5c5f2fcd817e467db8512b5a1dd9796b"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-k0b7"
server_key: "5722aea6bfdfd35aa2810ed3a3504bae"
server_key_iv: "68277c18bccab7040867e085aba78e52"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.578667 7483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "5c5f2fcd817e467db8512b5a1dd9796b"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-k0b7"
server_key: "5722aea6bfdfd35aa2810ed3a3504bae"
server_key_iv: "68277c18bccab7040867e085aba78e52"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.584751 7483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:00:55.589202 7497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.590935 7483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:00:55.591199 7483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "5c5f2fcd817e467db8512b5a1dd9796b"
format_stamp: "Formatted at 2026-09-29 01:00:55 on dist-test-slave-k0b7"
server_key: "5722aea6bfdfd35aa2810ed3a3504bae"
server_key_iv: "68277c18bccab7040867e085aba78e52"
server_key_version: "encryptionkey@0"
I20260929 01:00:55.591452 7483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:55.608354 7483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:55.613401 7483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:55.613740 7483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:55.614924 7483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:55.616408 7483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:55.616547 7483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.616698 7483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:55.616797 7483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:55.655812 7483 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:43937
I20260929 01:00:55.655912 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:43937 every 8 connection(s)
I20260929 01:00:55.656766 7483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:55.660880 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7483
I20260929 01:00:55.661103 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:55.661541 3159 external_mini_cluster.cc:1428] Setting key 7d08848c95f5f97088ab24f9897a6184
Sep 29 01:00:55 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:55.681399 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:55.659496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:37109 (local address 127.3.21.254:33369)
0929 01:00:55.660155 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:55.660160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:55.660907 (+ 747us) server_negotiation.cc:407] Connection header received
0929 01:00:55.663331 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:55.663335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:55.663399 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:55.663649 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:55.665518 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.666763 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.669009 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.669693 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.669804 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:55.674602 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:55.674607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:55.674611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:55.674648 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:55.677897 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:55.678932 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:55.678939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:55.678942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:55.679092 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:55.679546 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:55.679556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:55.679558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:55.679824 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:55.679998 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:55.680755 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 01:00:55.680993 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":455,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:00:55.682452 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:55.659976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:33369 (local address 127.3.21.194:37109)
0929 01:00:55.660724 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:55.660773 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:55.662985 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:55.663881 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:55.663895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:55.664510 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:55.665296 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:55.665317 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.666996 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.667001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:55.668790 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:55.668809 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:55.669987 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:55.669997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:55.670418 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:55.671319 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:55.671361 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:55.674280 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:55.678098 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:55.678109 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:55.678131 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:55.678780 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:55.679216 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:55.679222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:55.679226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:55.679415 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:55.679959 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:55.679968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:55.680369 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 01:00:55.680657 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 01:00:55.681216 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":275,"threads_started":1}
I20260929 01:00:55.684173 7610 heartbeater.cc:347] Connected to a master server at 127.3.21.254:33369
I20260929 01:00:55.684599 7610 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:00:55.685727 7610 heartbeater.cc:511] Master 127.3.21.254:33369 requested a full tablet report, sending...
I20260929 01:00:55.687619 7289 ts_manager.cc:194] Registered new tserver with Master: 5c5f2fcd817e467db8512b5a1dd9796b (127.3.21.194:43937)
I20260929 01:00:55.688702 7289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:37109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:00:55.715581 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:33369
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:00:56.087307 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:56.087726 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:56.087903 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:56.096257 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:56.096431 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:56.096694 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:56.107688 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:33369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:56.109133 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:56.110611 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:56.125728 7621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:56.126132 7624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:56.126644 7622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:56.126976 7616 server_base.cc:1034] running on GCE node
I20260929 01:00:56.127782 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:56.129206 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:56.130458 7616 hybrid_clock.cc:648] HybridClock initialized: now 1790643656130416 us; error 52 us; skew 500 ppm
I20260929 01:00:56.131013 7616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:00:56 dist-test-slave-k0b7 krb5kdc[7245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643656, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:00:56.135022 7616 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:00:56.136514 7616 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:56.136637 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:56.136981 7616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:56.140625 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "eee9420a22ce4f928f068240302ba5d7"
format_stamp: "Formatted at 2026-09-29 01:00:56 on dist-test-slave-k0b7"
server_key: "66da0108daf04db3b196c56de8956c3b"
server_key_iv: "19c9025850bd55dc441e6227de6fedc1"
server_key_version: "encryptionkey@0"
I20260929 01:00:56.141647 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "eee9420a22ce4f928f068240302ba5d7"
format_stamp: "Formatted at 2026-09-29 01:00:56 on dist-test-slave-k0b7"
server_key: "66da0108daf04db3b196c56de8956c3b"
server_key_iv: "19c9025850bd55dc441e6227de6fedc1"
server_key_version: "encryptionkey@0"
I20260929 01:00:56.145514 7477 heartbeater.cc:503] Master 127.3.21.254:33369 was elected leader, sending a full tablet report...
I20260929 01:00:56.148227 7616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 01:00:56.152211 7630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:56.153970 7616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:00:56.154134 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "eee9420a22ce4f928f068240302ba5d7"
format_stamp: "Formatted at 2026-09-29 01:00:56 on dist-test-slave-k0b7"
server_key: "66da0108daf04db3b196c56de8956c3b"
server_key_iv: "19c9025850bd55dc441e6227de6fedc1"
server_key_version: "encryptionkey@0"
I20260929 01:00:56.154340 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:56.169162 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:56.173910 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:56.174161 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:56.175230 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:56.176733 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:56.176823 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:56.176898 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:56.176954 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:56.213150 7616 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:40371
I20260929 01:00:56.213207 7742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:40371 every 8 connection(s)
I20260929 01:00:56.214048 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:56.219291 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7616
I20260929 01:00:56.219516 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:56.219985 3159 external_mini_cluster.cc:1428] Setting key 4cf02b22f0da67999bbcef47c2bf4611
Sep 29 01:00:56 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643656, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:00:56.239876 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.216870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:34563 (local address 127.3.21.254:33369)
0929 01:00:56.217354 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.217359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.218467 (+ 1108us) server_negotiation.cc:407] Connection header received
0929 01:00:56.220824 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.220832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.220961 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.221090 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:56.223511 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.224672 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.226888 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.227331 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.227443 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.231885 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:56.231890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:56.231894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:56.231944 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:56.236028 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:56.237250 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:56.237256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:56.237261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:56.237413 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:56.237887 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:56.237897 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:56.237899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:56.238192 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:56.238375 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.239221 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.239467 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260929 01:00:56.241014 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.217516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:33369 (local address 127.3.21.195:34563)
0929 01:00:56.218278 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:56.218326 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:56.220386 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:56.221485 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:56.221502 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:56.222283 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:56.223289 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.223317 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.224944 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.224949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.226664 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.226682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.227615 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.227622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.227870 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.228731 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:00:56.228762 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:56.231574 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:56.236383 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:56.236394 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:56.236414 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:56.237098 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:56.237548 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:00:56.237558 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:00:56.237560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:00:56.237756 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:00:56.238338 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:56.238347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:56.238817 (+ 470us) client_negotiation.cc:769] Sending connection context
0929 01:00:56.239109 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 01:00:56.239717 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260929 01:00:56.243096 7743 heartbeater.cc:347] Connected to a master server at 127.3.21.254:33369
I20260929 01:00:56.243502 7743 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:56.244197 7743 heartbeater.cc:511] Master 127.3.21.254:33369 requested a full tablet report, sending...
I20260929 01:00:56.245877 7289 ts_manager.cc:194] Registered new tserver with Master: eee9420a22ce4f928f068240302ba5d7 (127.3.21.195:40371)
I20260929 01:00:56.246928 7289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:34563
I20260929 01:00:56.250358 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:56.269730 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.255664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.3.21.254:33369)
0929 01:00:56.255862 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.255866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.256444 (+ 578us) server_negotiation.cc:407] Connection header received
0929 01:00:56.256720 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.256724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.256792 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.256904 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:56.258623 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.259338 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.260669 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.261140 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.261295 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.264038 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:56.264045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:56.264049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:56.264090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:56.266890 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:56.267659 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:56.267664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:56.267668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:56.267797 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:56.268177 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:56.268183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:56.268188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:56.268552 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:56.268827 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.269157 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.269389 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:00:56.274688 7289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:56.282884 7289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:56.308990 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.300078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.3.21.195:40371)
0929 01:00:56.301259 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.301268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.301299 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:00:56.301421 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.301428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.301677 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.301867 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:56.302815 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.303987 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.306070 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.307511 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.307683 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.307943 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.308406 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.308619 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":868,"threads_started":1}
I20260929 01:00:56.309067 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.299587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:40371 (local address 127.0.0.1:53046)
0929 01:00:56.300121 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:56.300137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:56.300272 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:56.302107 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:56.302112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:56.302136 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:56.302598 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.302610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.304136 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.304149 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.305805 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.305826 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.307737 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.307743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.307976 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.308040 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 01:00:56.308353 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 01:00:56.308645 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":9,"thread_start_us":655,"threads_started":1}
I20260929 01:00:56.309443 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.297344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.3.21.193:40241)
0929 01:00:56.298559 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.298569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.299202 (+ 633us) server_negotiation.cc:407] Connection header received
0929 01:00:56.299674 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.299685 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.299988 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.300162 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:56.301309 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.302765 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.306050 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.307690 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.307830 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.308047 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.308926 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.309113 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":748,"threads_started":1}
I20260929 01:00:56.309830 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.297852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43937 (local address 127.0.0.1:57332)
0929 01:00:56.299509 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:56.299526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:56.299736 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:56.300475 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:56.300482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:56.300527 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:56.301182 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.301200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.303233 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.303239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.305836 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.305851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.308391 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.308396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.308649 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.308740 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 01:00:56.308834 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 01:00:56.308988 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"mutex_wait_us":202,"spinlock_wait_cycles":1280,"thread_start_us":1356,"threads_started":1}
I20260929 01:00:56.310019 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.298522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.3.21.194:43937)
0929 01:00:56.299067 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.299075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.299618 (+ 543us) server_negotiation.cc:407] Connection header received
0929 01:00:56.299944 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.299955 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.300207 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.300367 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:56.301403 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.303050 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.306064 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.307684 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.307826 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.308017 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.309490 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.309688 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
I20260929 01:00:56.310928 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.296986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:40241 (local address 127.0.0.1:36588)
0929 01:00:56.299003 (+ 2017us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:56.299059 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:56.299391 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:56.300475 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:56.300482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:56.300534 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:56.301101 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.301120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.302926 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.302930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.305835 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.305851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.308383 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.308391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.308652 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.308739 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 01:00:56.310335 (+ 1596us) client_negotiation.cc:240] Negotiation successful
0929 01:00:56.310452 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1632,"mutex_wait_us":164,"spinlock_wait_cycles":17152,"thread_start_us":1518,"threads_started":1}
I20260929 01:00:56.314622 7545 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0ba1106f8f47e7929e18784d5ef802 (DEFAULT_TABLE table=test-table [id=0017d9a0f43a42cb9ea8c1e6a802b41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:56.314615 7678 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0ba1106f8f47e7929e18784d5ef802 (DEFAULT_TABLE table=test-table [id=0017d9a0f43a42cb9ea8c1e6a802b41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:56.316186 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0ba1106f8f47e7929e18784d5ef802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:56.316190 7545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0ba1106f8f47e7929e18784d5ef802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:56.317529 7412 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0ba1106f8f47e7929e18784d5ef802 (DEFAULT_TABLE table=test-table [id=0017d9a0f43a42cb9ea8c1e6a802b41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:56.319020 7412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0ba1106f8f47e7929e18784d5ef802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:56.332597 7763 tablet_bootstrap.cc:492] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Bootstrap starting.
I20260929 01:00:56.338096 7763 tablet_bootstrap.cc:654] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:56.340368 7763 log.cc:824] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:56.339481 7762 tablet_bootstrap.cc:492] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Bootstrap starting.
I20260929 01:00:56.341115 7764 tablet_bootstrap.cc:492] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Bootstrap starting.
I20260929 01:00:56.344693 7762 tablet_bootstrap.cc:654] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:56.344883 7764 tablet_bootstrap.cc:654] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:56.345114 7763 tablet_bootstrap.cc:492] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: No bootstrap required, opened a new log
I20260929 01:00:56.345453 7763 ts_tablet_manager.cc:1397] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260929 01:00:56.345956 7762 log.cc:824] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:56.346177 7764 log.cc:824] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:56.349434 7764 tablet_bootstrap.cc:492] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: No bootstrap required, opened a new log
I20260929 01:00:56.349565 7762 tablet_bootstrap.cc:492] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: No bootstrap required, opened a new log
I20260929 01:00:56.349785 7764 ts_tablet_manager.cc:1397] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 01:00:56.349843 7762 ts_tablet_manager.cc:1397] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260929 01:00:56.356776 7763 raft_consensus.cc:348] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.357410 7763 raft_consensus.cc:374] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:56.357543 7763 raft_consensus.cc:729] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5f2fcd817e467db8512b5a1dd9796b, State: Initialized, Role: FOLLOWER
I20260929 01:00:56.358327 7764 raft_consensus.cc:348] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.358675 7763 consensus_queue.cc:260] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.358893 7764 raft_consensus.cc:374] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:56.359037 7764 raft_consensus.cc:729] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee9420a22ce4f928f068240302ba5d7, State: Initialized, Role: FOLLOWER
I20260929 01:00:56.359697 7764 consensus_queue.cc:260] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.360033 7610 heartbeater.cc:503] Master 127.3.21.254:33369 was elected leader, sending a full tablet report...
I20260929 01:00:56.360782 7763 ts_tablet_manager.cc:1428] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 01:00:56.361099 7762 raft_consensus.cc:348] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.361563 7762 raft_consensus.cc:374] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:56.361666 7762 raft_consensus.cc:729] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7263b2cc2c344c08bc7f905ccde42511, State: Initialized, Role: FOLLOWER
I20260929 01:00:56.362102 7743 heartbeater.cc:503] Master 127.3.21.254:33369 was elected leader, sending a full tablet report...
I20260929 01:00:56.362609 7764 ts_tablet_manager.cc:1428] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 01:00:56.363736 7762 consensus_queue.cc:260] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.365685 7762 ts_tablet_manager.cc:1428] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Time spent starting tablet: real 0.016s user 0.011s sys 0.002s
W20260929 01:00:56.408937 7611 tablet.cc:2369] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:56.465763 7744 tablet.cc:2369] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:00:56.607812 7478 tablet.cc:2369] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:56.647923 7770 raft_consensus.cc:482] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:56.648236 7770 raft_consensus.cc:504] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.650405 7770 leader_election.cc:288] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee9420a22ce4f928f068240302ba5d7 (127.3.21.195:40371), 5c5f2fcd817e467db8512b5a1dd9796b (127.3.21.194:43937)
I20260929 01:00:56.660331 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.651018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:40371 (local address 127.3.21.193:51951)
0929 01:00:56.651871 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:56.651886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:56.652118 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:56.652501 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:56.652507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:56.652532 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:56.653013 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.653027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.654633 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.654639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.657331 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.657352 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.659248 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.659257 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.659511 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.659554 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:00:56.659678 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 01:00:56.659850 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":97,"thread_start_us":722,"threads_started":1}
I20260929 01:00:56.660434 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.651065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:51951 (local address 127.3.21.195:40371)
0929 01:00:56.651288 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.651295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.652057 (+ 762us) server_negotiation.cc:407] Connection header received
0929 01:00:56.652282 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.652290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.652361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.652461 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:56.653170 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.654429 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.657561 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.658951 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.659153 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.659229 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.659743 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.659899 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:00:56.661522 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0ba1106f8f47e7929e18784d5ef802" candidate_uuid: "7263b2cc2c344c08bc7f905ccde42511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee9420a22ce4f928f068240302ba5d7" is_pre_election: true
I20260929 01:00:56.661748 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.651177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:56485 (local address 127.3.21.194:43937)
0929 01:00:56.651440 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.651447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.652242 (+ 795us) server_negotiation.cc:407] Connection header received
0929 01:00:56.652427 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.652432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.652505 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.652621 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:00:56.653358 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.654429 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.658435 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.659942 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.660114 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.660185 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.661314 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.661482 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:00:56.662125 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.651049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:43937 (local address 127.3.21.193:56485)
0929 01:00:56.652081 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:56.652132 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:56.652291 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:56.652641 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:56.652646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:56.652689 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:00:56.653120 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.653130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.654634 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.654639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.657331 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:00:56.657352 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.660367 (+ 3015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.660375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:00:56.661108 (+ 733us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.661149 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:00:56.661265 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:00:56.661418 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":72,"thread_start_us":390,"threads_started":1}
I20260929 01:00:56.662101 7698 raft_consensus.cc:2454] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7263b2cc2c344c08bc7f905ccde42511 in term 0.
I20260929 01:00:56.662875 7366 leader_election.cc:302] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7263b2cc2c344c08bc7f905ccde42511, eee9420a22ce4f928f068240302ba5d7; no voters:
I20260929 01:00:56.663311 7770 raft_consensus.cc:2781] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:56.663487 7770 raft_consensus.cc:482] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:56.663288 7565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0ba1106f8f47e7929e18784d5ef802" candidate_uuid: "7263b2cc2c344c08bc7f905ccde42511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" is_pre_election: true
I20260929 01:00:56.663610 7770 raft_consensus.cc:3037] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:56.663844 7565 raft_consensus.cc:2454] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7263b2cc2c344c08bc7f905ccde42511 in term 0.
I20260929 01:00:56.666196 7770 raft_consensus.cc:504] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.666973 7770 leader_election.cc:288] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [CANDIDATE]: Term 1 election: Requested vote from peers eee9420a22ce4f928f068240302ba5d7 (127.3.21.195:40371), 5c5f2fcd817e467db8512b5a1dd9796b (127.3.21.194:43937)
I20260929 01:00:56.667481 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0ba1106f8f47e7929e18784d5ef802" candidate_uuid: "7263b2cc2c344c08bc7f905ccde42511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee9420a22ce4f928f068240302ba5d7"
I20260929 01:00:56.667554 7565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0ba1106f8f47e7929e18784d5ef802" candidate_uuid: "7263b2cc2c344c08bc7f905ccde42511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c5f2fcd817e467db8512b5a1dd9796b"
I20260929 01:00:56.667790 7565 raft_consensus.cc:3037] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:56.667763 7698 raft_consensus.cc:3037] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:56.671526 7565 raft_consensus.cc:2454] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7263b2cc2c344c08bc7f905ccde42511 in term 1.
I20260929 01:00:56.671517 7698 raft_consensus.cc:2454] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7263b2cc2c344c08bc7f905ccde42511 in term 1.
I20260929 01:00:56.672035 7366 leader_election.cc:302] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7263b2cc2c344c08bc7f905ccde42511, eee9420a22ce4f928f068240302ba5d7; no voters:
I20260929 01:00:56.672446 7770 raft_consensus.cc:2781] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:56.672904 7770 raft_consensus.cc:686] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 1 LEADER]: Becoming Leader. State: Replica: 7263b2cc2c344c08bc7f905ccde42511, State: Running, Role: LEADER
I20260929 01:00:56.673422 7770 consensus_queue.cc:237] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } }
I20260929 01:00:56.679311 7289 catalog_manager.cc:5495] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7263b2cc2c344c08bc7f905ccde42511 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "7263b2cc2c344c08bc7f905ccde42511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7263b2cc2c344c08bc7f905ccde42511" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:56.776633 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.768499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.3.21.193:40241)
0929 01:00:56.768692 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.768699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.768824 (+ 125us) server_negotiation.cc:407] Connection header received
0929 01:00:56.769157 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.769165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.769292 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.769468 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:00:56.770355 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.771544 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.774069 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.774599 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.774759 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.775335 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:00:56.775941 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:00:56.776203 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.776255 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.776347 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:00:56.787079 7698 raft_consensus.cc:1261] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 1 FOLLOWER]: Refusing update from remote peer 7263b2cc2c344c08bc7f905ccde42511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:56.787072 7565 raft_consensus.cc:1261] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 1 FOLLOWER]: Refusing update from remote peer 7263b2cc2c344c08bc7f905ccde42511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:56.788578 7773 consensus_queue.cc:1035] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c5f2fcd817e467db8512b5a1dd9796b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:56.789142 7770 consensus_queue.cc:1035] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee9420a22ce4f928f068240302ba5d7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:56.802052 7776 mvcc.cc:205] Tried to move back new op lower bound from 7334476418187493376 to 7334476417738452992. Current Snapshot: MvccSnapshot[applied={T|T < 7334476418187493376}]
I20260929 01:00:56.818105 7289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:56.818454 7289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:56.822239 7289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1166c2a0e5e34cfaa22437825b1da122: Sending DeleteTablet for 3 replicas of tablet 1b0ba1106f8f47e7929e18784d5ef802
I20260929 01:00:56.823736 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0ba1106f8f47e7929e18784d5ef802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:56 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:53046
I20260929 01:00:56.824156 7545 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0ba1106f8f47e7929e18784d5ef802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:56 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:57332
I20260929 01:00:56.824566 7412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b0ba1106f8f47e7929e18784d5ef802 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:56 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:36588
I20260929 01:00:56.825212 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7350
I20260929 01:00:56.835395 7787 tablet_replica.cc:330] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: stopping tablet replica
I20260929 01:00:56.836186 7787 raft_consensus.cc:2229] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:56.836944 7787 raft_consensus.cc:2258] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:56.838128 7786 tablet_replica.cc:330] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: stopping tablet replica
I20260929 01:00:56.838701 7788 tablet_replica.cc:330] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: stopping tablet replica
I20260929 01:00:56.838873 7786 raft_consensus.cc:2229] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:56.839294 7788 raft_consensus.cc:2229] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:56.839500 7786 raft_consensus.cc:2258] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:56.840152 7788 raft_consensus.cc:2258] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:56.842072 7786 ts_tablet_manager.cc:1905] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:56.842435 7788 ts_tablet_manager.cc:1905] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:56.847734 7787 ts_tablet_manager.cc:1905] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:56.851262 7786 ts_tablet_manager.cc:1918] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:56.851562 7786 log.cc:1196] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/1b0ba1106f8f47e7929e18784d5ef802
I20260929 01:00:56.852246 7786 ts_tablet_manager.cc:1939] T 1b0ba1106f8f47e7929e18784d5ef802 P eee9420a22ce4f928f068240302ba5d7: Deleting consensus metadata
I20260929 01:00:56.853904 7787 ts_tablet_manager.cc:1918] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:56.854094 7787 log.cc:1196] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/1b0ba1106f8f47e7929e18784d5ef802
I20260929 01:00:56.855027 7787 ts_tablet_manager.cc:1939] T 1b0ba1106f8f47e7929e18784d5ef802 P 5c5f2fcd817e467db8512b5a1dd9796b: Deleting consensus metadata
I20260929 01:00:56.857272 7276 catalog_manager.cc:4841] TS 5c5f2fcd817e467db8512b5a1dd9796b (127.3.21.194:43937): tablet 1b0ba1106f8f47e7929e18784d5ef802 (table test-table [id=0017d9a0f43a42cb9ea8c1e6a802b41e]) successfully deleted
Sep 29 01:00:56 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:00:56.859210 7274 catalog_manager.cc:4841] TS eee9420a22ce4f928f068240302ba5d7 (127.3.21.195:40371): tablet 1b0ba1106f8f47e7929e18784d5ef802 (table test-table [id=0017d9a0f43a42cb9ea8c1e6a802b41e]) successfully deleted
I20260929 01:00:56.860496 7788 ts_tablet_manager.cc:1918] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:56.860694 7788 log.cc:1196] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/1b0ba1106f8f47e7929e18784d5ef802
I20260929 01:00:56.861253 7788 ts_tablet_manager.cc:1939] T 1b0ba1106f8f47e7929e18784d5ef802 P 7263b2cc2c344c08bc7f905ccde42511: Deleting consensus metadata
I20260929 01:00:56.863404 7274 catalog_manager.cc:4841] TS 7263b2cc2c344c08bc7f905ccde42511 (127.3.21.193:40241): tablet 1b0ba1106f8f47e7929e18784d5ef802 (table test-table [id=0017d9a0f43a42cb9ea8c1e6a802b41e]) successfully deleted
I20260929 01:00:56.868641 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.827099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.3.21.193:40241)
0929 01:00:56.827330 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.827339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.837651 (+ 10312us) server_negotiation.cc:407] Connection header received
0929 01:00:56.837746 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.837750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.837820 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.837957 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:56.839782 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.841109 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.843359 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.844213 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.844352 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:56.860022 (+ 15670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:56.860044 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:56.860073 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:56.860138 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:56.864788 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:56.865937 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:56.865946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:56.865949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:56.866238 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:56.866902 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:56.866907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:56.866909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:56.867318 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:56.867580 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:56.867972 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 01:00:56.868232 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":73}
I20260929 01:00:56.869297 7472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:56.975116 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7350
I20260929 01:00:56.993763 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7483
Sep 29 01:00:56 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:00:57.010987 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:56.994803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.3.21.194:43937)
0929 01:00:56.995037 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:56.995043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:56.995067 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:00:56.995119 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:56.995123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:56.995194 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:56.995305 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:56.996601 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.997354 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.998690 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:56.999102 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:56.999247 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:57.003881 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:57.003898 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:57.003915 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:57.003970 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:57.007786 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:57.008654 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:57.008661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:57.008664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:57.008808 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:57.009232 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:57.009236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:57.009238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:57.009683 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:57.010028 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:57.010358 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 01:00:57.010635 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:00:57.011991 7605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:57.109874 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7483
I20260929 01:00:57.130141 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7616
Sep 29 01:00:57 dist-test-slave-k0b7 krb5kdc[7245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:00:57.150717 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:57.131349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.3.21.195:40371)
0929 01:00:57.131785 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:57.131790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:57.131813 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:00:57.131880 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:57.131884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:57.132055 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:57.132174 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:57.133705 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:57.134579 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:57.136062 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:00:57.136515 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:00:57.136696 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:00:57.142236 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:57.142254 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:57.142281 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:00:57.142326 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:57.146873 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:57.148011 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:57.148021 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:57.148023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:57.148230 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:00:57.148827 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:00:57.148831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:00:57.148833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:00:57.149295 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:57.149663 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:57.149999 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 01:00:57.150214 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20260929 01:00:57.151844 7738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:57.246730 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7616
I20260929 01:00:57.265136 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7258
I20260929 01:00:57.266707 7319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:57.377521 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7258
2026-09-29T01:00:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4798 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T01:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:00:57Z Disabled control of system clock
I20260929 01:00:57.452756 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:44937
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 01:00:57.813678 7797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:57.814072 7797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:57.814193 7797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:57.821648 7797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:00:57.821786 7797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:57.821823 7797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:57.821872 7797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:00:57.821916 7797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:00:57.832368 7797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:44937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:57.833861 7797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:57.835758 7797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:57.848235 7802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:57.848248 7805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:57.848307 7803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:57.849028 7797 server_base.cc:1034] running on GCE node
I20260929 01:00:57.849910 7797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:57.851778 7797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:57.853004 7797 hybrid_clock.cc:648] HybridClock initialized: now 1790643657852966 us; error 37 us; skew 500 ppm
I20260929 01:00:57.853403 7797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:00:57.856128 7797 webserver.cc:460] Webserver started at http://127.3.21.254:35025/ using document root <none> and password file <none>
I20260929 01:00:57.856894 7797 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:57.856983 7797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:57.857366 7797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:57.861670 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "29d66177be404353aca9dc28692dae43"
format_stamp: "Formatted at 2026-09-29 01:00:57 on dist-test-slave-k0b7"
server_key: "37b018b6aee12cfef29731a916faee6f"
server_key_iv: "c5f829e477a80e0da7d41be57ccd94aa"
server_key_version: "encryptionkey@0"
I20260929 01:00:57.862699 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "29d66177be404353aca9dc28692dae43"
format_stamp: "Formatted at 2026-09-29 01:00:57 on dist-test-slave-k0b7"
server_key: "37b018b6aee12cfef29731a916faee6f"
server_key_iv: "c5f829e477a80e0da7d41be57ccd94aa"
server_key_version: "encryptionkey@0"
I20260929 01:00:57.868450 7797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:00:57.872843 7811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:57.874397 7797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:00:57.874658 7797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "29d66177be404353aca9dc28692dae43"
format_stamp: "Formatted at 2026-09-29 01:00:57 on dist-test-slave-k0b7"
server_key: "37b018b6aee12cfef29731a916faee6f"
server_key_iv: "c5f829e477a80e0da7d41be57ccd94aa"
server_key_version: "encryptionkey@0"
I20260929 01:00:57.874922 7797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:57.881278 7797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:57.882182 7797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:57.882577 7797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:57.902814 7797 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:44937
I20260929 01:00:57.902833 7862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:44937 every 8 connection(s)
I20260929 01:00:57.904305 7797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:00:57.906248 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7797
I20260929 01:00:57.906639 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:00:57.907294 3159 external_mini_cluster.cc:1428] Setting key 1d9a329c84cb06d4d8bd1b833cd0c445
I20260929 01:00:57.909026 7863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:57.921725 7863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43: Bootstrap starting.
I20260929 01:00:57.923385 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:57.909397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.3.21.254:44937)
0929 01:00:57.910651 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:57.910667 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:57.910723 (+ 56us) server_negotiation.cc:407] Connection header received
0929 01:00:57.912271 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:57.912293 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:57.912737 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:57.913187 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:57.921368 (+ 8181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:57.921375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:57.921392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:57.921418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:57.921503 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:57.921522 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:57.921792 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:57.921943 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 01:00:57.922366 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":420,"threads_started":1}
I20260929 01:00:57.925486 7863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:57.926726 7863 log.cc:824] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:57.929935 7863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43: No bootstrap required, opened a new log
I20260929 01:00:57.937793 7863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d66177be404353aca9dc28692dae43" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 44937 } }
I20260929 01:00:57.938191 7863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:57.938269 7863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d66177be404353aca9dc28692dae43, State: Initialized, Role: FOLLOWER
I20260929 01:00:57.938875 7863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d66177be404353aca9dc28692dae43" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 44937 } }
I20260929 01:00:57.939091 7863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:00:57.939194 7863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:00:57.939357 7863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:57.941507 7863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d66177be404353aca9dc28692dae43" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 44937 } }
I20260929 01:00:57.942060 7863 leader_election.cc:302] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d66177be404353aca9dc28692dae43; no voters:
I20260929 01:00:57.942817 7863 leader_election.cc:288] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:00:57.943150 7868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:57.944252 7868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [term 1 LEADER]: Becoming Leader. State: Replica: 29d66177be404353aca9dc28692dae43, State: Running, Role: LEADER
I20260929 01:00:57.944861 7868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d66177be404353aca9dc28692dae43" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 44937 } }
I20260929 01:00:57.945426 7863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:00:57.948206 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d66177be404353aca9dc28692dae43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d66177be404353aca9dc28692dae43" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 44937 } } }
I20260929 01:00:57.948216 7870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d66177be404353aca9dc28692dae43. Latest consensus state: current_term: 1 leader_uuid: "29d66177be404353aca9dc28692dae43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d66177be404353aca9dc28692dae43" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 44937 } } }
I20260929 01:00:57.948930 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:57.949010 7870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43 [sys.catalog]: This master's current role is: LEADER
I20260929 01:00:57.951416 7874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:00:57.956571 7874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:00:57.965197 7874 catalog_manager.cc:1347] Generated new cluster ID: 0da3ab27c5e94440ab35939a5fb6ebd3
I20260929 01:00:57.965332 7874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:00:57.987134 7874 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:00:57.987689 7874 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:00:58.003232 7874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43: Generated new TSK 0
I20260929 01:00:58.004498 7874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:00:58.015906 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:44937
--builtin_ntp_servers=127.3.21.212:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 01:00:58.404633 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:58.404968 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:58.405169 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:58.412842 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:58.412972 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:58.413095 7887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:00:58.422889 7887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:58.424290 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:58.426203 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:58.441380 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:58.441632 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:58.441489 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:58.442463 7887 server_base.cc:1034] running on GCE node
I20260929 01:00:58.443799 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:58.444959 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:58.446290 7887 hybrid_clock.cc:648] HybridClock initialized: now 1790643658446201 us; error 74 us; skew 500 ppm
I20260929 01:00:58.447012 7887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:00:58.450178 7887 webserver.cc:460] Webserver started at http://127.3.21.193:38419/ using document root <none> and password file <none>
I20260929 01:00:58.451166 7887 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:58.451289 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:58.451759 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:58.455762 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "fc016e209de34de6901fd644204b68fb"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-k0b7"
server_key: "38ddee16a24122bf53d126a664b55f84"
server_key_iv: "b30684753f05e0c326f954cd154f17da"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.456789 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "fc016e209de34de6901fd644204b68fb"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-k0b7"
server_key: "38ddee16a24122bf53d126a664b55f84"
server_key_iv: "b30684753f05e0c326f954cd154f17da"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.464228 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:00:58.469261 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:58.471338 7887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:00:58.471594 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "fc016e209de34de6901fd644204b68fb"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-k0b7"
server_key: "38ddee16a24122bf53d126a664b55f84"
server_key_iv: "b30684753f05e0c326f954cd154f17da"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.471885 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:58.478788 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:58.479835 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:58.480144 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:58.481139 7887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:58.482715 7887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:58.482815 7887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:58.482894 7887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:58.482949 7887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:58.519935 7887 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:37099
I20260929 01:00:58.520048 8013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:37099 every 8 connection(s)
I20260929 01:00:58.521353 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:00:58.528920 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 7887
I20260929 01:00:58.529214 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:00:58.529932 3159 external_mini_cluster.cc:1428] Setting key 12f7c43c886b089579fb0c8c4e9f75ae
I20260929 01:00:58.530926 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:58.524287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:43031 (local address 127.3.21.254:44937)
0929 01:00:58.524875 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:58.524879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:58.526294 (+ 1415us) server_negotiation.cc:407] Connection header received
0929 01:00:58.528207 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:58.528212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:58.528295 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:58.528419 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:58.530052 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:58.530058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:58.530064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:58.530088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:58.530143 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:58.530158 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:58.530295 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:58.530557 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 01:00:58.530632 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":360,"threads_started":1}
I20260929 01:00:58.532099 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:58.525117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:44937 (local address 127.3.21.193:43031)
0929 01:00:58.526098 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:58.526137 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:58.527903 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:58.528706 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:58.528716 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:58.528820 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:58.529714 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:58.529717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:58.529758 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:58.529851 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:58.530255 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:58.530265 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:58.530273 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:00:58.530633 (+ 360us) client_negotiation.cc:240] Negotiation successful
0929 01:00:58.530948 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260929 01:00:58.533136 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:44937
--builtin_ntp_servers=127.3.21.212:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 01:00:58.533875 8014 heartbeater.cc:347] Connected to a master server at 127.3.21.254:44937
I20260929 01:00:58.534235 8014 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:58.535039 8014 heartbeater.cc:511] Master 127.3.21.254:44937 requested a full tablet report, sending...
I20260929 01:00:58.537595 7828 ts_manager.cc:194] Registered new tserver with Master: fc016e209de34de6901fd644204b68fb (127.3.21.193:37099)
W20260929 01:00:58.898072 8018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:58.898397 8018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:58.898564 8018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:58.909101 8018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:58.909267 8018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:58.909456 8018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:00:58.925716 8018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:58.927544 8018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:58.929392 8018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:58.944306 8023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:58.944341 8024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:58.944401 8026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:58.945358 8018 server_base.cc:1034] running on GCE node
I20260929 01:00:58.946255 8018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:58.947573 8018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:58.948843 8018 hybrid_clock.cc:648] HybridClock initialized: now 1790643658948808 us; error 67 us; skew 500 ppm
I20260929 01:00:58.949218 8018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:00:58.952100 8018 webserver.cc:460] Webserver started at http://127.3.21.194:39269/ using document root <none> and password file <none>
I20260929 01:00:58.952873 8018 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:58.953014 8018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:58.953356 8018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:58.956481 8018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-k0b7"
server_key: "4c24d14050b566eb3726f7c9e6d72a67"
server_key_iv: "479f440bdf581d838c3bf58b984b8a19"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.957381 8018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-k0b7"
server_key: "4c24d14050b566eb3726f7c9e6d72a67"
server_key_iv: "479f440bdf581d838c3bf58b984b8a19"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.963290 8018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:00:58.967396 8032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:58.969319 8018 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:00:58.969491 8018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6"
format_stamp: "Formatted at 2026-09-29 01:00:58 on dist-test-slave-k0b7"
server_key: "4c24d14050b566eb3726f7c9e6d72a67"
server_key_iv: "479f440bdf581d838c3bf58b984b8a19"
server_key_version: "encryptionkey@0"
I20260929 01:00:58.969714 8018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:58.976197 8018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:58.977190 8018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:58.977461 8018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:58.978600 8018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:58.980187 8018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:58.980286 8018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:58.980365 8018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:58.980423 8018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:59.019233 8018 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:40357
I20260929 01:00:59.019322 8144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:40357 every 8 connection(s)
I20260929 01:00:59.020640 8018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:00:59.027019 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8018
I20260929 01:00:59.027231 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:00:59.027752 3159 external_mini_cluster.cc:1428] Setting key 660efb6a7a9f4cc11d0cdde3ccfd004d
I20260929 01:00:59.030431 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.023838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:40329 (local address 127.3.21.254:44937)
0929 01:00:59.024061 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.024067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.025761 (+ 1694us) server_negotiation.cc:407] Connection header received
0929 01:00:59.027714 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.027720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.027802 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.027921 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.029560 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.029572 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.029575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.029591 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.029622 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.029657 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.029797 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.030170 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.030224 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 01:00:59.031138 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:44937
--builtin_ntp_servers=127.3.21.212:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 01:00:59.032056 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.024467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:44937 (local address 127.3.21.194:40329)
0929 01:00:59.025560 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.025613 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.027413 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.028250 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.028264 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.028428 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.029230 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.029233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.029276 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.029374 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.029790 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.029799 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.029806 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.030273 (+ 467us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.030611 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":322,"threads_started":1}
I20260929 01:00:59.034052 8145 heartbeater.cc:347] Connected to a master server at 127.3.21.254:44937
I20260929 01:00:59.034412 8145 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:59.035256 8145 heartbeater.cc:511] Master 127.3.21.254:44937 requested a full tablet report, sending...
I20260929 01:00:59.037180 7828 ts_manager.cc:194] Registered new tserver with Master: 7c415e34c56b4e0a962ae8b4c21e1fd6 (127.3.21.194:40357)
W20260929 01:00:59.389170 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:00:59.389504 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:00:59.389650 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:00:59.397462 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:00:59.397595 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:00:59.397792 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:00:59.408411 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:00:59.409878 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:00:59.411947 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:00:59.427634 8153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:59.427783 8156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:00:59.427804 8154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:00:59.429114 8148 server_base.cc:1034] running on GCE node
I20260929 01:00:59.429795 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:00:59.431021 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:00:59.432264 8148 hybrid_clock.cc:648] HybridClock initialized: now 1790643659432184 us; error 65 us; skew 500 ppm
I20260929 01:00:59.432816 8148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:00:59.435860 8148 webserver.cc:460] Webserver started at http://127.3.21.195:33121/ using document root <none> and password file <none>
I20260929 01:00:59.436677 8148 fs_manager.cc:359] Metadata directory not provided
I20260929 01:00:59.436772 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:00:59.437145 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:00:59.440335 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc"
format_stamp: "Formatted at 2026-09-29 01:00:59 on dist-test-slave-k0b7"
server_key: "4d98d88c4410eeab804bd0b816daa30c"
server_key_iv: "d8a5728acf6a81046fcb8d1aee331673"
server_key_version: "encryptionkey@0"
I20260929 01:00:59.441231 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc"
format_stamp: "Formatted at 2026-09-29 01:00:59 on dist-test-slave-k0b7"
server_key: "4d98d88c4410eeab804bd0b816daa30c"
server_key_iv: "d8a5728acf6a81046fcb8d1aee331673"
server_key_version: "encryptionkey@0"
I20260929 01:00:59.447764 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:00:59.452013 8162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:59.453722 8148 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:00:59.453917 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc"
format_stamp: "Formatted at 2026-09-29 01:00:59 on dist-test-slave-k0b7"
server_key: "4d98d88c4410eeab804bd0b816daa30c"
server_key_iv: "d8a5728acf6a81046fcb8d1aee331673"
server_key_version: "encryptionkey@0"
I20260929 01:00:59.454192 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:00:59.460397 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:00:59.461364 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:00:59.461665 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:00:59.462843 8148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:00:59.464428 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:00:59.464520 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:59.464684 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:00:59.464751 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:00:59.500316 8148 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:43235
I20260929 01:00:59.500383 8274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:43235 every 8 connection(s)
I20260929 01:00:59.501786 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:00:59.506809 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8148
I20260929 01:00:59.507006 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:00:59.507752 3159 external_mini_cluster.cc:1428] Setting key 67b2f2a66e3ac481aa61fa923cf08926
I20260929 01:00:59.513293 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.505481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:52267 (local address 127.3.21.254:44937)
0929 01:00:59.505773 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.505778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.508055 (+ 2277us) server_negotiation.cc:407] Connection header received
0929 01:00:59.510520 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.510526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.510601 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.510719 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.512406 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.512411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.512415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.512433 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.512465 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.512478 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.512589 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.513005 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.513064 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 01:00:59.514580 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.506197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:44937 (local address 127.3.21.195:52267)
0929 01:00:59.507836 (+ 1639us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.507891 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.510170 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.510966 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.510982 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.511175 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.512063 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.512066 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.512113 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.512208 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.512619 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.512630 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.512800 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.513074 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.513324 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":558,"threads_started":1}
I20260929 01:00:59.516754 8275 heartbeater.cc:347] Connected to a master server at 127.3.21.254:44937
I20260929 01:00:59.517144 8275 heartbeater.cc:464] Registering TS with master...
I20260929 01:00:59.518033 8275 heartbeater.cc:511] Master 127.3.21.254:44937 requested a full tablet report, sending...
I20260929 01:00:59.520031 7828 ts_manager.cc:194] Registered new tserver with Master: a75ab9f60dac4a8d8dd3ef634a0ab5cc (127.3.21.195:43235)
I20260929 01:00:59.524048 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:00:59.531737 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.528964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.3.21.254:44937)
0929 01:00:59.529235 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.529244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.529671 (+ 427us) server_negotiation.cc:407] Connection header received
0929 01:00:59.529877 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.529882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.529945 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.530061 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.531080 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.531085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.531090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.531108 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.531148 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.531160 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.531290 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.531385 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.531448 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:00:59.540998 8014 heartbeater.cc:503] Master 127.3.21.254:44937 was elected leader, sending a full tablet report...
I20260929 01:00:59.536114 7828 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:00:59.543617 7828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:00:59.564013 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.557732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.3.21.193:37099)
0929 01:00:59.558580 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.558592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.560130 (+ 1538us) server_negotiation.cc:407] Connection header received
0929 01:00:59.560362 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.560370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.560564 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.560687 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.562946 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.562956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.562972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.562993 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.563061 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.563078 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.563246 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.563521 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.563624 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":365,"threads_started":1}
I20260929 01:00:59.564081 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.557958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:43235 (local address 127.0.0.1:55970)
0929 01:00:59.558972 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.559041 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.559256 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.561345 (+ 2089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.561352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.561445 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.562625 (+ 1180us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.562629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.562666 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.562750 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.563299 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.563311 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.563320 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.563507 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.563624 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":622,"threads_started":1}
I20260929 01:00:59.565044 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.558010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.3.21.195:43235)
0929 01:00:59.559051 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.559061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.559351 (+ 290us) server_negotiation.cc:407] Connection header received
0929 01:00:59.559446 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.559454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.559696 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.560182 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.562946 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.562956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.562972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.562992 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.563055 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.563076 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.563246 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.564514 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.564639 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":729,"threads_started":1}
I20260929 01:00:59.565773 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.557959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36814)
0929 01:00:59.560008 (+ 2049us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.560043 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.560211 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.561675 (+ 1464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.561681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.561712 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.562672 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.562674 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.562686 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.562750 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.563299 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.563311 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.563320 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.565282 (+ 1962us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.565337 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"mutex_wait_us":362,"thread_start_us":1476,"threads_started":1}
I20260929 01:00:59.568287 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.560366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.3.21.194:40357)
0929 01:00:59.565742 (+ 5376us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.565753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.565789 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:00:59.565864 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.565876 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.566148 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.566305 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.567467 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.567479 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.567494 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.567513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.567577 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.567599 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.567747 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.567857 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.567965 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5196,"thread_start_us":5060,"threads_started":1}
I20260929 01:00:59.568334 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.560017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:40357 (local address 127.0.0.1:52946)
0929 01:00:59.564899 (+ 4882us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.564924 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.565023 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.566403 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.566408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.566431 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.567229 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.567233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.567250 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.567294 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.567724 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.567729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.567734 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.567874 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.567928 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4728,"thread_start_us":4581,"threads_started":1}
I20260929 01:00:59.569636 8210 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffbdccaf19940d39ba7ab8816eeac6f (DEFAULT_TABLE table=test-table [id=4c687db3e5a64e779a5957650fb710ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:59.569944 7949 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffbdccaf19940d39ba7ab8816eeac6f (DEFAULT_TABLE table=test-table [id=4c687db3e5a64e779a5957650fb710ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:59.571142 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffbdccaf19940d39ba7ab8816eeac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:59.571149 7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffbdccaf19940d39ba7ab8816eeac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:59.583693 8080 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffbdccaf19940d39ba7ab8816eeac6f (DEFAULT_TABLE table=test-table [id=4c687db3e5a64e779a5957650fb710ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:00:59.584486 8293 tablet_bootstrap.cc:492] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: Bootstrap starting.
I20260929 01:00:59.585609 8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffbdccaf19940d39ba7ab8816eeac6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:00:59.588794 8293 tablet_bootstrap.cc:654] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:59.589754 8294 tablet_bootstrap.cc:492] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Bootstrap starting.
I20260929 01:00:59.590198 8293 log.cc:824] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:59.594076 8294 tablet_bootstrap.cc:654] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:59.594175 8293 tablet_bootstrap.cc:492] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: No bootstrap required, opened a new log
I20260929 01:00:59.594815 8293 ts_tablet_manager.cc:1397] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 01:00:59.595592 8294 log.cc:824] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:59.599534 8294 tablet_bootstrap.cc:492] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: No bootstrap required, opened a new log
I20260929 01:00:59.599858 8294 ts_tablet_manager.cc:1397] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 01:00:59.601632 8297 tablet_bootstrap.cc:492] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Bootstrap starting.
I20260929 01:00:59.606129 8297 tablet_bootstrap.cc:654] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Neither blocks nor log segments found. Creating new log.
I20260929 01:00:59.605904 8293 raft_consensus.cc:348] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.606820 8293 raft_consensus.cc:374] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:59.606959 8293 raft_consensus.cc:729] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc016e209de34de6901fd644204b68fb, State: Initialized, Role: FOLLOWER
I20260929 01:00:59.607635 8297 log.cc:824] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:00:59.607699 8293 consensus_queue.cc:260] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.610121 8293 ts_tablet_manager.cc:1428] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 01:00:59.611584 8294 raft_consensus.cc:348] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.611999 8294 raft_consensus.cc:374] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:59.612138 8294 raft_consensus.cc:729] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75ab9f60dac4a8d8dd3ef634a0ab5cc, State: Initialized, Role: FOLLOWER
I20260929 01:00:59.612779 8294 consensus_queue.cc:260] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.613979 8275 heartbeater.cc:503] Master 127.3.21.254:44937 was elected leader, sending a full tablet report...
I20260929 01:00:59.614711 8294 ts_tablet_manager.cc:1428] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260929 01:00:59.614924 8297 tablet_bootstrap.cc:492] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: No bootstrap required, opened a new log
I20260929 01:00:59.615386 8297 ts_tablet_manager.cc:1397] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260929 01:00:59.623457 8297 raft_consensus.cc:348] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.623903 8297 raft_consensus.cc:374] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:00:59.624037 8297 raft_consensus.cc:729] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c415e34c56b4e0a962ae8b4c21e1fd6, State: Initialized, Role: FOLLOWER
I20260929 01:00:59.624682 8297 consensus_queue.cc:260] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.625965 8145 heartbeater.cc:503] Master 127.3.21.254:44937 was elected leader, sending a full tablet report...
I20260929 01:00:59.626586 8297 ts_tablet_manager.cc:1428] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260929 01:00:59.685669 8300 raft_consensus.cc:482] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:00:59.685986 8300 raft_consensus.cc:504] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.687796 8300 leader_election.cc:288] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc016e209de34de6901fd644204b68fb (127.3.21.193:37099), 7c415e34c56b4e0a962ae8b4c21e1fd6 (127.3.21.194:40357)
I20260929 01:00:59.691620 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.688298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:41269 (local address 127.3.21.194:40357)
0929 01:00:59.688561 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.688568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.689248 (+ 680us) server_negotiation.cc:407] Connection header received
0929 01:00:59.689331 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.689340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.689410 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.689499 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.690897 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.690903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.690908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.690944 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.690998 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.691019 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.691182 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.691238 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.691304 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 01:00:59.691639 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.688757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:40357 (local address 127.3.21.195:41269)
0929 01:00:59.688968 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.688991 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.689112 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.689747 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.689750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.689800 (+ 50us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.690670 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.690672 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.690689 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.690726 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.691145 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.691149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.691153 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.691235 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.691277 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 01:00:59.692080 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.688442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:47435 (local address 127.3.21.193:37099)
0929 01:00:59.688632 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.688642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.689819 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 01:00:59.690072 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.690078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.690137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.690212 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.691449 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.691453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.691457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.691476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.691513 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.691527 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.691658 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.691790 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.691853 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:00:59.692308 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.688853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.3.21.195:47435)
0929 01:00:59.689712 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 01:00:59.689730 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:00:59.689918 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:00:59.690279 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:00:59.690286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:00:59.690316 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:00:59.691230 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:00:59.691233 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:00:59.691250 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:00:59.691292 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:00:59.691652 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:00:59.691657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:00:59.691664 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:00:59.691801 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 01:00:59.691862 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":474,"threads_started":1}
I20260929 01:00:59.693064 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffbdccaf19940d39ba7ab8816eeac6f" candidate_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" is_pre_election: true
I20260929 01:00:59.693411 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffbdccaf19940d39ba7ab8816eeac6f" candidate_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc016e209de34de6901fd644204b68fb" is_pre_election: true
I20260929 01:00:59.693608 8100 raft_consensus.cc:2454] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a75ab9f60dac4a8d8dd3ef634a0ab5cc in term 0.
I20260929 01:00:59.693922 7969 raft_consensus.cc:2454] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a75ab9f60dac4a8d8dd3ef634a0ab5cc in term 0.
I20260929 01:00:59.694531 8165 leader_election.cc:302] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c415e34c56b4e0a962ae8b4c21e1fd6, a75ab9f60dac4a8d8dd3ef634a0ab5cc; no voters:
I20260929 01:00:59.694959 8300 raft_consensus.cc:2781] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:00:59.695123 8300 raft_consensus.cc:482] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:00:59.695225 8300 raft_consensus.cc:3037] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:59.697562 8300 raft_consensus.cc:504] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.698230 8300 leader_election.cc:288] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [CANDIDATE]: Term 1 election: Requested vote from peers fc016e209de34de6901fd644204b68fb (127.3.21.193:37099), 7c415e34c56b4e0a962ae8b4c21e1fd6 (127.3.21.194:40357)
I20260929 01:00:59.698855 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffbdccaf19940d39ba7ab8816eeac6f" candidate_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6"
I20260929 01:00:59.698858 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffbdccaf19940d39ba7ab8816eeac6f" candidate_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc016e209de34de6901fd644204b68fb"
I20260929 01:00:59.699159 8100 raft_consensus.cc:3037] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:59.699187 7969 raft_consensus.cc:3037] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:00:59.701647 8100 raft_consensus.cc:2454] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a75ab9f60dac4a8d8dd3ef634a0ab5cc in term 1.
I20260929 01:00:59.701648 7969 raft_consensus.cc:2454] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a75ab9f60dac4a8d8dd3ef634a0ab5cc in term 1.
I20260929 01:00:59.702239 8165 leader_election.cc:302] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7c415e34c56b4e0a962ae8b4c21e1fd6, a75ab9f60dac4a8d8dd3ef634a0ab5cc, fc016e209de34de6901fd644204b68fb; no voters:
I20260929 01:00:59.702732 8300 raft_consensus.cc:2781] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:00:59.703459 8300 raft_consensus.cc:686] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 1 LEADER]: Becoming Leader. State: Replica: a75ab9f60dac4a8d8dd3ef634a0ab5cc, State: Running, Role: LEADER
I20260929 01:00:59.704276 8300 consensus_queue.cc:237] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } }
I20260929 01:00:59.709687 7827 catalog_manager.cc:5495] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc reported cstate change: term changed from 0 to 1, leader changed from <none> to a75ab9f60dac4a8d8dd3ef634a0ab5cc (127.3.21.195). New cstate: current_term: 1 leader_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a75ab9f60dac4a8d8dd3ef634a0ab5cc" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 43235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 } health_report { overall_health: UNKNOWN } } }
I20260929 01:00:59.733176 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.729993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.3.21.195:43235)
0929 01:00:59.730297 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.730303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.730331 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:00:59.730659 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.730664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.730757 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.730864 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.732343 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.732348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.732352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.732371 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.732406 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.732419 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.732599 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.732794 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.732849 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260929 01:00:59.743404 8100 raft_consensus.cc:1261] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 1 FOLLOWER]: Refusing update from remote peer a75ab9f60dac4a8d8dd3ef634a0ab5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:59.743909 7969 raft_consensus.cc:1261] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 1 FOLLOWER]: Refusing update from remote peer a75ab9f60dac4a8d8dd3ef634a0ab5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:00:59.745142 8303 consensus_queue.cc:1035] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c415e34c56b4e0a962ae8b4c21e1fd6" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:00:59.745687 8300 consensus_queue.cc:1035] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc016e209de34de6901fd644204b68fb" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:00:59.754045 8276 tablet.cc:2369] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:00:59.782614 7827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:59.782950 7827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:00:59.787490 7827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29d66177be404353aca9dc28692dae43: Sending DeleteTablet for 3 replicas of tablet 8ffbdccaf19940d39ba7ab8816eeac6f
I20260929 01:00:59.788944 7949 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffbdccaf19940d39ba7ab8816eeac6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:59 UTC) from {username='slave'} at 127.0.0.1:36814
I20260929 01:00:59.789052 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffbdccaf19940d39ba7ab8816eeac6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:59 UTC) from {username='slave'} at 127.0.0.1:55970
I20260929 01:00:59.789330 8080 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffbdccaf19940d39ba7ab8816eeac6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:00:59 UTC) from {username='slave'} at 127.0.0.1:52946
I20260929 01:00:59.791926 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7887
I20260929 01:00:59.815572 8318 tablet_replica.cc:330] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: stopping tablet replica
I20260929 01:00:59.816617 8316 tablet_replica.cc:330] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: stopping tablet replica
I20260929 01:00:59.816870 8317 tablet_replica.cc:330] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: stopping tablet replica
I20260929 01:00:59.817071 8318 raft_consensus.cc:2229] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:59.817251 8317 raft_consensus.cc:2229] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:00:59.817305 8316 raft_consensus.cc:2229] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:00:59.817549 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:00:59.794653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.3.21.193:37099)
0929 01:00:59.795195 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 01:00:59.795207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:00:59.813118 (+ 17911us) server_negotiation.cc:407] Connection header received
0929 01:00:59.814331 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:00:59.814341 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:00:59.814545 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:00:59.814706 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:00:59.816480 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:00:59.816487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:00:59.816494 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:00:59.816529 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:00:59.816578 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:00:59.816599 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:00:59.816743 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:00:59.817056 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 01:00:59.817137 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 01:00:59.817696 8318 raft_consensus.cc:2258] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:59.817806 8317 raft_consensus.cc:2258] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:59.818784 8009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:00:59.819223 8316 raft_consensus.cc:2258] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:00:59.821029 8318 ts_tablet_manager.cc:1905] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:59.821805 8316 ts_tablet_manager.cc:1905] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:59.830338 8316 ts_tablet_manager.cc:1918] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:59.830634 8316 log.cc:1196] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/8ffbdccaf19940d39ba7ab8816eeac6f
I20260929 01:00:59.831250 8316 ts_tablet_manager.cc:1939] T 8ffbdccaf19940d39ba7ab8816eeac6f P a75ab9f60dac4a8d8dd3ef634a0ab5cc: Deleting consensus metadata
I20260929 01:00:59.834138 7812 catalog_manager.cc:4841] TS a75ab9f60dac4a8d8dd3ef634a0ab5cc (127.3.21.195:43235): tablet 8ffbdccaf19940d39ba7ab8816eeac6f (table test-table [id=4c687db3e5a64e779a5957650fb710ad]) successfully deleted
I20260929 01:00:59.836694 8318 ts_tablet_manager.cc:1918] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:00:59.837023 8318 log.cc:1196] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/8ffbdccaf19940d39ba7ab8816eeac6f
I20260929 01:00:59.837611 8318 ts_tablet_manager.cc:1939] T 8ffbdccaf19940d39ba7ab8816eeac6f P 7c415e34c56b4e0a962ae8b4c21e1fd6: Deleting consensus metadata
I20260929 01:00:59.839609 7814 catalog_manager.cc:4841] TS 7c415e34c56b4e0a962ae8b4c21e1fd6 (127.3.21.194:40357): tablet 8ffbdccaf19940d39ba7ab8816eeac6f (table test-table [id=4c687db3e5a64e779a5957650fb710ad]) successfully deleted
I20260929 01:00:59.980324 8317 ts_tablet_manager.cc:1905] T 8ffbdccaf19940d39ba7ab8816eeac6f P fc016e209de34de6901fd644204b68fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:00:59.983064 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7887
I20260929 01:01:00.003314 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8018
W20260929 01:01:00.003971 7815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.21.193:37099 (error 108)
I20260929 01:01:00.006091 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.005452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36838)
0929 01:01:00.005711 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.005881 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260929 01:01:00.006397 7815 catalog_manager.cc:4584] TS fc016e209de34de6901fd644204b68fb (127.3.21.193:37099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ffbdccaf19940d39ba7ab8816eeac6f: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
I20260929 01:01:00.007526 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.004503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.3.21.194:40357)
0929 01:01:00.004714 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:00.004722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:00.005378 (+ 656us) server_negotiation.cc:407] Connection header received
0929 01:01:00.005575 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:00.005580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:00.005697 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:00.005826 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:00.006868 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:00.006878 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:00.006881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:00.006899 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:00.006932 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:00.006944 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:00.007092 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:00.007174 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 01:01:00.007243 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:01:00.008374 8140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:00.060286 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.059712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36848)
0929 01:01:00.059963 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.060083 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 01:01:00.061625 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.061185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36850)
0929 01:01:00.061365 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.061463 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:01:00.104883 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8018
I20260929 01:01:00.123301 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.122066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36866)
0929 01:01:00.122358 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.122746 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 01:01:00.124302 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8148
I20260929 01:01:00.125200 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.124709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36876)
0929 01:01:00.124895 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.125005 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 01:01:00.129487 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.125437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.3.21.195:43235)
0929 01:01:00.125629 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:00.125635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:00.125661 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:01:00.126236 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:00.126242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:00.126335 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:00.126466 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:00.128422 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:00.128432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:00.128447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:00.128480 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:00.128551 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:00.128582 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:00.128827 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:00.129039 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 01:01:00.129129 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:01:00.130407 8270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:00.206787 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.206118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36880)
0929 01:01:00.206420 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.206600 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 01:01:00.208096 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.207628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36882)
0929 01:01:00.207860 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.207940 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 01:01:00.262598 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8148
I20260929 01:01:00.282331 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 7797
I20260929 01:01:00.283800 7858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:00.398653 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.396981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:37099 (local address 127.0.0.1:36894)
0929 01:01:00.398085 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:00.398231 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:37099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":838}
I20260929 01:01:00.398882 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 7797
2026-09-29T01:01:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3021 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 01:01:00.445556 3159 mini_oidc.cc:152] Starting JWKS server
I20260929 01:01:00.448357 3159 webserver.cc:460] Webserver started at https://127.0.0.1:44587/ using document root <none> and password file <none>
I20260929 01:01:00.448728 3159 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:01:00.449463 3159 webserver.cc:460] Webserver started at http://127.0.0.1:35551/ using document root <none> and password file <none>
2026-09-29T01:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:00Z Disabled control of system clock
I20260929 01:01:00.480757 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36051
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36051
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44587/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/testchainca.cert with env {}
W20260929 01:01:00.832350 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:00.832737 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:00.832904 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:00.841436 8330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:00.841612 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:00.841706 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:00.841820 8330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:00.841897 8330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:01:00.841992 8330 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:01:00.842072 8330 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44587/jwks.json
I20260929 01:01:00.853345 8330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44587/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:00.854844 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:00.856201 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:00.869138 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:00.869071 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:00.869237 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:00.870505 8330 server_base.cc:1034] running on GCE node
I20260929 01:01:00.871732 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:00.873831 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:00.875193 8330 hybrid_clock.cc:648] HybridClock initialized: now 1790643660875092 us; error 77 us; skew 500 ppm
I20260929 01:01:00.875743 8330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:00.879012 8330 webserver.cc:460] Webserver started at http://127.3.21.254:35593/ using document root <none> and password file <none>
I20260929 01:01:00.879841 8330 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:00.879989 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:00.880373 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:00.883767 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "2fa7aa08944248598fc658984cb084e0"
format_stamp: "Formatted at 2026-09-29 01:01:00 on dist-test-slave-k0b7"
server_key: "2cf4b5e01db7c99955dabe5758f944f4"
server_key_iv: "9a511fcbcaca1e2d8dccd26307a09ac9"
server_key_version: "encryptionkey@0"
I20260929 01:01:00.884716 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "2fa7aa08944248598fc658984cb084e0"
format_stamp: "Formatted at 2026-09-29 01:01:00 on dist-test-slave-k0b7"
server_key: "2cf4b5e01db7c99955dabe5758f944f4"
server_key_iv: "9a511fcbcaca1e2d8dccd26307a09ac9"
server_key_version: "encryptionkey@0"
I20260929 01:01:00.890614 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:01:00.894944 8344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:00.896746 8330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:01:00.896984 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "2fa7aa08944248598fc658984cb084e0"
format_stamp: "Formatted at 2026-09-29 01:01:00 on dist-test-slave-k0b7"
server_key: "2cf4b5e01db7c99955dabe5758f944f4"
server_key_iv: "9a511fcbcaca1e2d8dccd26307a09ac9"
server_key_version: "encryptionkey@0"
I20260929 01:01:00.897246 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:00.916201 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:00.917200 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:00.917557 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:00.939211 8330 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36051
I20260929 01:01:00.939246 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36051 every 8 connection(s)
I20260929 01:01:00.940898 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:00.943327 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8330
I20260929 01:01:00.943652 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:00.944293 3159 external_mini_cluster.cc:1428] Setting key 06de9fca379de3b37ff0947d72d36ede
I20260929 01:01:00.945746 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:00.958854 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:00.946385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.3.21.254:36051)
0929 01:01:00.947437 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:00.947448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:00.947493 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:01:00.949176 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:00.949202 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:00.949720 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:00.950210 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:00.950858 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:00.952274 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:00.953796 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:00.954593 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:00.954799 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:00.956153 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:00.956161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:00.956180 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:00.956820 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:00.956910 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:00.956941 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:00.957078 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:00.957330 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 01:01:00.957716 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":327,"threads_started":1}
I20260929 01:01:00.959003 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0: Bootstrap starting.
I20260929 01:01:00.962873 8396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:00.963768 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:00.964141 8396 log.cc:824] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:00.967298 8396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0: No bootstrap required, opened a new log
I20260929 01:01:00.975322 8396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7aa08944248598fc658984cb084e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36051 } }
I20260929 01:01:00.975773 8396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:00.975889 8396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa7aa08944248598fc658984cb084e0, State: Initialized, Role: FOLLOWER
I20260929 01:01:00.976671 8396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7aa08944248598fc658984cb084e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36051 } }
I20260929 01:01:00.976938 8396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:00.977089 8396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:00.977272 8396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:00.979595 8396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7aa08944248598fc658984cb084e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36051 } }
I20260929 01:01:00.980167 8396 leader_election.cc:302] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa7aa08944248598fc658984cb084e0; no voters:
I20260929 01:01:00.980813 8396 leader_election.cc:288] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:00.980997 8401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:00.982339 8401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa7aa08944248598fc658984cb084e0, State: Running, Role: LEADER
I20260929 01:01:00.982913 8401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7aa08944248598fc658984cb084e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36051 } }
I20260929 01:01:00.983211 8396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:00.985973 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa7aa08944248598fc658984cb084e0. Latest consensus state: current_term: 1 leader_uuid: "2fa7aa08944248598fc658984cb084e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7aa08944248598fc658984cb084e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36051 } } }
I20260929 01:01:00.986404 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:00.985920 8402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa7aa08944248598fc658984cb084e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7aa08944248598fc658984cb084e0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36051 } } }
I20260929 01:01:00.987013 8402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:00.988929 8409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:00.994470 8409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:01.007261 8409 catalog_manager.cc:1347] Generated new cluster ID: 273845e6575148c18c38c7272340e0e6
I20260929 01:01:01.007407 8409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:01.028617 8409 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:01.030624 8409 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:01.039757 8409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fa7aa08944248598fc658984cb084e0: Generated new TSK 0
I20260929 01:01:01.041020 8409 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 01:01:01.096419 8426 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 01:01:01.096628 8426 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:01:01.097561 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:01.065627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.3.21.254:36051)
0929 01:01:01.065915 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:01.065925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:01.066687 (+ 762us) server_negotiation.cc:407] Connection header received
0929 01:01:01.066944 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:01.066949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:01.067034 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:01.067179 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:01:01.067883 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.068689 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.071056 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.071533 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.071756 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:01.072524 (+ 768us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:01:01.096959 (+ 24435us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 01:01:01.097283 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36422: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 01:01:01.097739 8399 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36422: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 01:01:01.099680 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:01.065394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36051 (local address 127.0.0.1:36422)
0929 01:01:01.066317 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:01.066340 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:01.066643 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:01.067238 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:01.067245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:01.067259 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:01:01.067675 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:01.067688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.068851 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.068857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:01.070781 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:01.070831 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.071837 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.071845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:01.072050 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:01.072148 (+ 98us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:01:01.097299 (+ 25151us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 01:01:01.097719 (+ 420us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.21.254:36051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":543,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.21.254:36051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 01:01:01.101935 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8330
I20260929 01:01:01.103590 8391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:01.201311 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8330
2026-09-29T01:01:01Z chronyd exiting
I20260929 01:01:01.254222 3159 test_util.cc:175] -----------------------------------------------
I20260929 01:01:01.254432 3159 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643611838815-3159-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (813 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 01:01:01.258817 3159 mini_oidc.cc:152] Starting JWKS server
I20260929 01:01:01.260922 3159 webserver.cc:460] Webserver started at https://127.0.0.1:34825/ using document root <none> and password file <none>
I20260929 01:01:01.261207 3159 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:01:01.261979 3159 webserver.cc:460] Webserver started at http://127.0.0.1:37179/ using document root <none> and password file <none>
2026-09-29T01:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:01Z Disabled control of system clock
I20260929 01:01:01.293471 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34101
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34101
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34825/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/testchainca.cert with env {}
W20260929 01:01:01.667047 8433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:01.667362 8433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:01.667523 8433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:01.675112 8433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:01.675243 8433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:01.675282 8433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:01.675369 8433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:01.675438 8433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:01:01.675498 8433 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:01:01.675576 8433 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34825/jwks.json
I20260929 01:01:01.686928 8433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34825/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:01.688380 8433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:01.689772 8433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:01.703145 8438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:01.703184 8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:01.703220 8439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:01.704351 8433 server_base.cc:1034] running on GCE node
I20260929 01:01:01.705479 8433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:01.707199 8433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:01.708652 8433 hybrid_clock.cc:648] HybridClock initialized: now 1790643661708609 us; error 57 us; skew 500 ppm
I20260929 01:01:01.709129 8433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:01.711936 8433 webserver.cc:460] Webserver started at http://127.3.21.254:43615/ using document root <none> and password file <none>
I20260929 01:01:01.712783 8433 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:01.712937 8433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:01.713304 8433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:01.716570 8433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "8ed90a12a37b470595adb5901b5ee550"
format_stamp: "Formatted at 2026-09-29 01:01:01 on dist-test-slave-k0b7"
server_key: "4d7724184f19285a054607f8cbf039b6"
server_key_iv: "55534eb690caab8ab50c2725136ae23b"
server_key_version: "encryptionkey@0"
I20260929 01:01:01.717521 8433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "8ed90a12a37b470595adb5901b5ee550"
format_stamp: "Formatted at 2026-09-29 01:01:01 on dist-test-slave-k0b7"
server_key: "4d7724184f19285a054607f8cbf039b6"
server_key_iv: "55534eb690caab8ab50c2725136ae23b"
server_key_version: "encryptionkey@0"
I20260929 01:01:01.724422 8433 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 01:01:01.729213 8447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:01.731400 8433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 01:01:01.731680 8433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "8ed90a12a37b470595adb5901b5ee550"
format_stamp: "Formatted at 2026-09-29 01:01:01 on dist-test-slave-k0b7"
server_key: "4d7724184f19285a054607f8cbf039b6"
server_key_iv: "55534eb690caab8ab50c2725136ae23b"
server_key_version: "encryptionkey@0"
I20260929 01:01:01.731962 8433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:01.752251 8433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:01.753182 8433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:01.753535 8433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:01.775611 8433 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34101
I20260929 01:01:01.775663 8498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34101 every 8 connection(s)
I20260929 01:01:01.777285 8433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:01.778761 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8433
I20260929 01:01:01.778965 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:01.779748 3159 external_mini_cluster.cc:1428] Setting key 675d0e32653302702f6c2dd2e1da139c
I20260929 01:01:01.782827 8499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:01.794443 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:01.781843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.3.21.254:34101)
0929 01:01:01.783164 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:01.783176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:01.783223 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:01:01.784816 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:01.784839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:01.785354 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:01.785831 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:01.786435 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.787990 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.789383 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.790124 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.790338 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:01.791838 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:01.791847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:01.791871 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:01.792497 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:01.792590 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:01.792616 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:01.792773 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:01.793043 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 01:01:01.793413 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":395,"threads_started":1}
I20260929 01:01:01.795269 8499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550: Bootstrap starting.
I20260929 01:01:01.799731 8499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:01.799786 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:01.800899 8499 log.cc:824] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:01.804852 8499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550: No bootstrap required, opened a new log
I20260929 01:01:01.814006 8499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed90a12a37b470595adb5901b5ee550" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34101 } }
I20260929 01:01:01.814636 8499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:01.814777 8499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed90a12a37b470595adb5901b5ee550, State: Initialized, Role: FOLLOWER
I20260929 01:01:01.815575 8499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed90a12a37b470595adb5901b5ee550" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34101 } }
I20260929 01:01:01.815824 8499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:01.815970 8499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:01.816145 8499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:01.818795 8499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed90a12a37b470595adb5901b5ee550" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34101 } }
I20260929 01:01:01.819496 8499 leader_election.cc:302] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed90a12a37b470595adb5901b5ee550; no voters:
I20260929 01:01:01.820320 8499 leader_election.cc:288] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:01.820797 8504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:01.822168 8504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed90a12a37b470595adb5901b5ee550, State: Running, Role: LEADER
I20260929 01:01:01.822906 8499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:01.823302 8504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed90a12a37b470595adb5901b5ee550" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34101 } }
I20260929 01:01:01.827702 8505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed90a12a37b470595adb5901b5ee550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed90a12a37b470595adb5901b5ee550" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34101 } } }
I20260929 01:01:01.828228 8505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:01.827720 8506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed90a12a37b470595adb5901b5ee550. Latest consensus state: current_term: 1 leader_uuid: "8ed90a12a37b470595adb5901b5ee550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed90a12a37b470595adb5901b5ee550" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34101 } } }
I20260929 01:01:01.829978 8506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:01.831151 8514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:01.837325 8514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:01.849490 8514 catalog_manager.cc:1347] Generated new cluster ID: 1d085b5b44724b849160b7478ce49ece
I20260929 01:01:01.849664 8514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:01.876127 8514 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:01.879271 8514 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:01.897698 8514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ed90a12a37b470595adb5901b5ee550: Generated new TSK 0
I20260929 01:01:01.898842 8514 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 01:01:01.949597 8529 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 01:01:01.949745 8529 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:01:01.951068 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:01.929023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.3.21.254:34101)
0929 01:01:01.929310 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:01.929317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:01.929706 (+ 389us) server_negotiation.cc:407] Connection header received
0929 01:01:01.930088 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:01.930093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:01.930192 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:01.930343 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:01:01.930961 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.931885 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.934096 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.934879 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.935021 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:01.935778 (+ 757us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:01:01.950310 (+ 14532us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 01:01:01.950778 (+ 468us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":90}
W20260929 01:01:01.951252 8502 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 01:01:01.951412 8528 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:01.928787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34101 (local address 127.0.0.1:47196)
0929 01:01:01.929557 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:01.929579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:01.929763 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:01.930397 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:01.930402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:01.930416 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:01:01.930757 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:01.930770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.932054 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.932061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:01.933841 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:01.933864 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:01.935114 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:01.935122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:01.935353 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:01.935412 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:01:01.950654 (+ 15242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 01:01:01.950972 (+ 318us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.21.254:34101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":444,"threads_started":1}
I20260929 01:01:01.953342 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8433
I20260929 01:01:01.955651 8494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:02.057416 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8433
2026-09-29T01:01:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (857 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 01:01:02.116355 3159 mini_oidc.cc:152] Starting JWKS server
I20260929 01:01:02.118450 3159 webserver.cc:460] Webserver started at https://127.0.0.1:39735/ using document root <none> and password file <none>
I20260929 01:01:02.118831 3159 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:01:02.119791 3159 webserver.cc:460] Webserver started at http://127.0.0.1:45213/ using document root <none> and password file <none>
2026-09-29T01:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:02Z Disabled control of system clock
I20260929 01:01:02.155578 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:42363
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:42363
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39735/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 01:01:02.512737 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:02.513128 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:02.513325 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:02.521037 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:02.521230 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:02.521299 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:02.521387 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:02.521473 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:01:02.521564 8536 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:01:02.521638 8536 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39735/jwks.json
I20260929 01:01:02.531734 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:42363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39735/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:02.533171 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:02.535136 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:02.549237 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:02.549154 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:02.549206 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:02.550386 8536 server_base.cc:1034] running on GCE node
I20260929 01:01:02.551481 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:02.553184 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:02.554596 8536 hybrid_clock.cc:648] HybridClock initialized: now 1790643662554407 us; error 118 us; skew 500 ppm
I20260929 01:01:02.555094 8536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:02.564922 8536 webserver.cc:460] Webserver started at http://127.3.21.254:44229/ using document root <none> and password file <none>
I20260929 01:01:02.565814 8536 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:02.565994 8536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:02.566366 8536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:02.569737 8536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "c4f6ab9231664f8f83e07770c644806b"
format_stamp: "Formatted at 2026-09-29 01:01:02 on dist-test-slave-k0b7"
server_key: "ba5fca4a375438c94a9573461770abf1"
server_key_iv: "7f884fe7956d2cd7420ce67cf9ba77ac"
server_key_version: "encryptionkey@0"
I20260929 01:01:02.570793 8536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "c4f6ab9231664f8f83e07770c644806b"
format_stamp: "Formatted at 2026-09-29 01:01:02 on dist-test-slave-k0b7"
server_key: "ba5fca4a375438c94a9573461770abf1"
server_key_iv: "7f884fe7956d2cd7420ce67cf9ba77ac"
server_key_version: "encryptionkey@0"
I20260929 01:01:02.576839 8536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:01:02.581264 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:02.583314 8536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:01:02.583674 8536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "c4f6ab9231664f8f83e07770c644806b"
format_stamp: "Formatted at 2026-09-29 01:01:02 on dist-test-slave-k0b7"
server_key: "ba5fca4a375438c94a9573461770abf1"
server_key_iv: "7f884fe7956d2cd7420ce67cf9ba77ac"
server_key_version: "encryptionkey@0"
I20260929 01:01:02.583964 8536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:02.609745 8536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:02.610742 8536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:02.611101 8536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:02.635180 8536 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:42363
I20260929 01:01:02.635211 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:42363 every 8 connection(s)
I20260929 01:01:02.636881 8536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:02.640400 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8536
I20260929 01:01:02.640666 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:02.641230 8602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:02.641353 3159 external_mini_cluster.cc:1428] Setting key 9075e0601d7e12e360bf596c3d5a81db
I20260929 01:01:02.656183 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b: Bootstrap starting.
I20260929 01:01:02.657186 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:02.643412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.3.21.254:42363)
0929 01:01:02.644633 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:02.644645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:02.644694 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:01:02.646461 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:02.646598 (+ 137us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:02.647179 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:02.647645 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:02.648242 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.649761 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.651757 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.652536 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.652785 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:02.654242 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:02.654251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:02.654269 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:02.655032 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:02.655117 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:02.655141 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:02.655283 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:02.655526 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 01:01:02.655871 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":402,"threads_started":1}
I20260929 01:01:02.660274 8602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:02.661558 8602 log.cc:824] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:02.662177 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:02.665118 8602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b: No bootstrap required, opened a new log
I20260929 01:01:02.674958 8602 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f6ab9231664f8f83e07770c644806b" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42363 } }
I20260929 01:01:02.675527 8602 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:02.675657 8602 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6ab9231664f8f83e07770c644806b, State: Initialized, Role: FOLLOWER
I20260929 01:01:02.676388 8602 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f6ab9231664f8f83e07770c644806b" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42363 } }
I20260929 01:01:02.676671 8602 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:02.676842 8602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:02.677006 8602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:02.679478 8602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f6ab9231664f8f83e07770c644806b" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42363 } }
I20260929 01:01:02.680160 8602 leader_election.cc:302] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f6ab9231664f8f83e07770c644806b; no voters:
I20260929 01:01:02.680975 8602 leader_election.cc:288] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:02.681205 8607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:02.682113 8607 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [term 1 LEADER]: Becoming Leader. State: Replica: c4f6ab9231664f8f83e07770c644806b, State: Running, Role: LEADER
I20260929 01:01:02.682803 8607 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f6ab9231664f8f83e07770c644806b" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42363 } }
I20260929 01:01:02.683586 8602 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:02.685822 8609 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4f6ab9231664f8f83e07770c644806b. Latest consensus state: current_term: 1 leader_uuid: "c4f6ab9231664f8f83e07770c644806b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f6ab9231664f8f83e07770c644806b" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42363 } } }
I20260929 01:01:02.686214 8609 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:02.686718 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4f6ab9231664f8f83e07770c644806b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f6ab9231664f8f83e07770c644806b" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42363 } } }
I20260929 01:01:02.687077 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:02.688932 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:02.695466 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:02.705839 8614 catalog_manager.cc:1347] Generated new cluster ID: edddfc614c644abcb7ea374bbd38afc3
I20260929 01:01:02.705974 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:02.730393 8614 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:02.731853 8614 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:02.741361 8614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4f6ab9231664f8f83e07770c644806b: Generated new TSK 0
I20260929 01:01:02.742403 8614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:02.768250 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:02.756208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.3.21.254:42363)
0929 01:01:02.756430 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:02.756442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:02.757083 (+ 641us) server_negotiation.cc:407] Connection header received
0929 01:01:02.757309 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:02.757315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:02.757414 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:02.757642 (+ 228us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:01:02.758282 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.759527 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.761054 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.761763 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.761906 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:02.763123 (+ 1217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:01:02.767127 (+ 4004us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 01:01:02.767682 (+ 555us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 01:01:02.768460 8605 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 01:01:02.769748 8631 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:02.756141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42363 (local address 127.0.0.1:60644)
0929 01:01:02.756960 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:02.756977 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:02.757118 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:02.757724 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:02.757729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:02.757739 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:01:02.758112 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:02.758122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.759724 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.759729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:02.760873 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:02.760886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:02.762075 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:02.762087 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:02.762523 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:02.762589 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:01:02.768832 (+ 6243us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 01:01:02.769262 (+ 430us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.21.254:42363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":510,"threads_started":1}
I20260929 01:01:02.774600 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8536
I20260929 01:01:02.776139 8597 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 01:01:02.785310 8632 webserver.cc:533] Webserver: sslize failed:
E20260929 01:01:02.785430 8632 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:01:02.876662 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8536
2026-09-29T01:01:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (816 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T01:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:02Z Disabled control of system clock
I20260929 01:01:02.963874 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37097
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37097
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/testchainca.cert with env {}
W20260929 01:01:03.336619 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:03.336963 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:03.337059 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:03.344655 8637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:03.344796 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:03.344834 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:03.344884 8637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:03.344928 8637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:01:03.344990 8637 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:01:03.345023 8637 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 01:01:03.355636 8637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:03.357127 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:03.358517 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:03.376343 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:03.376448 8643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:03.376309 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:03.377527 8637 server_base.cc:1034] running on GCE node
I20260929 01:01:03.378607 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:03.380368 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:03.381672 8637 hybrid_clock.cc:648] HybridClock initialized: now 1790643663381622 us; error 54 us; skew 500 ppm
I20260929 01:01:03.382121 8637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:03.385439 8637 webserver.cc:460] Webserver started at http://127.3.21.254:37521/ using document root <none> and password file <none>
I20260929 01:01:03.386317 8637 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:03.386475 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:03.386904 8637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:03.390200 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "fd784b4f9ac5495cb1259b757784a225"
format_stamp: "Formatted at 2026-09-29 01:01:03 on dist-test-slave-k0b7"
server_key: "f4c37d3f3960ea83d98a8e85216a04cc"
server_key_iv: "3991017cd400fba755ed53f777fe2898"
server_key_version: "encryptionkey@0"
I20260929 01:01:03.391249 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "fd784b4f9ac5495cb1259b757784a225"
format_stamp: "Formatted at 2026-09-29 01:01:03 on dist-test-slave-k0b7"
server_key: "f4c37d3f3960ea83d98a8e85216a04cc"
server_key_iv: "3991017cd400fba755ed53f777fe2898"
server_key_version: "encryptionkey@0"
I20260929 01:01:03.398036 8637 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:01:03.402972 8651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:03.405260 8637 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:01:03.405632 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "fd784b4f9ac5495cb1259b757784a225"
format_stamp: "Formatted at 2026-09-29 01:01:03 on dist-test-slave-k0b7"
server_key: "f4c37d3f3960ea83d98a8e85216a04cc"
server_key_iv: "3991017cd400fba755ed53f777fe2898"
server_key_version: "encryptionkey@0"
I20260929 01:01:03.405931 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:03.427565 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:03.428606 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:03.428932 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:03.456549 8637 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:37097
I20260929 01:01:03.456571 8702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:37097 every 8 connection(s)
I20260929 01:01:03.459384 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:03.460427 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8637
I20260929 01:01:03.460865 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:03.462275 3159 external_mini_cluster.cc:1428] Setting key dee95715134ac0a9f3a0a4af0b402ee6
I20260929 01:01:03.465791 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:03.479017 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:03.464697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.3.21.254:37097)
0929 01:01:03.465809 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:03.465822 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:03.465902 (+ 80us) server_negotiation.cc:407] Connection header received
0929 01:01:03.467628 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:03.467651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:03.468181 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:03.468556 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:03.469530 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.471060 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.473170 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.474034 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.474186 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:03.475990 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:03.475999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:03.476017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:03.476693 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:03.476786 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:03.476811 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:03.476927 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:03.477279 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 01:01:03.477784 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":418,"threads_started":1}
I20260929 01:01:03.482707 8703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225: Bootstrap starting.
I20260929 01:01:03.484970 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:01:03.487093 8703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:03.488504 8703 log.cc:824] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:03.493167 8703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225: No bootstrap required, opened a new log
I20260929 01:01:03.502417 8703 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd784b4f9ac5495cb1259b757784a225" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37097 } }
I20260929 01:01:03.503008 8703 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:03.503136 8703 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd784b4f9ac5495cb1259b757784a225, State: Initialized, Role: FOLLOWER
I20260929 01:01:03.503827 8703 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd784b4f9ac5495cb1259b757784a225" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37097 } }
I20260929 01:01:03.504084 8703 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:03.504240 8703 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:03.504424 8703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:03.507153 8703 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd784b4f9ac5495cb1259b757784a225" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37097 } }
I20260929 01:01:03.507746 8703 leader_election.cc:302] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd784b4f9ac5495cb1259b757784a225; no voters:
I20260929 01:01:03.508424 8703 leader_election.cc:288] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:03.508730 8708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:03.509598 8708 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [term 1 LEADER]: Becoming Leader. State: Replica: fd784b4f9ac5495cb1259b757784a225, State: Running, Role: LEADER
I20260929 01:01:03.510236 8708 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd784b4f9ac5495cb1259b757784a225" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37097 } }
I20260929 01:01:03.510841 8703 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:03.513835 8709 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd784b4f9ac5495cb1259b757784a225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd784b4f9ac5495cb1259b757784a225" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37097 } } }
I20260929 01:01:03.514042 8710 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd784b4f9ac5495cb1259b757784a225. Latest consensus state: current_term: 1 leader_uuid: "fd784b4f9ac5495cb1259b757784a225" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd784b4f9ac5495cb1259b757784a225" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37097 } } }
I20260929 01:01:03.515563 8710 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:03.515576 8709 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:03.520272 8717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:03.526264 8717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:03.536412 8717 catalog_manager.cc:1347] Generated new cluster ID: 17b69c0e84194455bd19e0d2378b0f5c
I20260929 01:01:03.536561 8717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:03.554325 8717 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:03.556300 8717 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:03.568467 8717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd784b4f9ac5495cb1259b757784a225: Generated new TSK 0
I20260929 01:01:03.569530 8717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:04.092988 8706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:03.595051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.3.21.254:37097)
0929 01:01:03.595284 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:03.595292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:03.596123 (+ 831us) server_negotiation.cc:407] Connection header received
0929 01:01:03.596362 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:03.596370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:03.596499 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:03.596678 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:01:03.597259 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.598403 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.600095 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.600822 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.600957 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:03.601562 (+ 605us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:01:04.092352 (+490790us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 01:01:04.092756 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46962: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 01:01:04.093269 8732 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:03.594956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37097 (local address 127.0.0.1:46962)
0929 01:01:03.595994 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:03.596014 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:03.596178 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:03.596721 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:03.596725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:03.596736 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:01:03.597093 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:03.597104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.598703 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.598708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:03.599908 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:03.599923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:03.601036 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:03.601043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:03.601179 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:03.601218 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:01:04.092720 (+491502us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 01:01:04.092967 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.21.254:37097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":650,"threads_started":1}
W20260929 01:01:04.093502 8706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46962: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260929 01:01:04.094671 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8637
I20260929 01:01:04.096189 8698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:04.203104 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8637
2026-09-29T01:01:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1310 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (474 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1606 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:06 dist-test-slave-k0b7 krb5kdc[8748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:06 dist-test-slave-k0b7 krb5kdc[8748](info): set up 2 sockets
Sep 29 01:01:06 dist-test-slave-k0b7 krb5kdc[8748](info): commencing operation
krb5kdc: starting...
W20260929 01:01:07.459462 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:01:07 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:01:07.704717 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34001
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:08.081892 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:08.082242 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:08.082350 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:08.091034 8761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:08.091178 8761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:08.091231 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:08.091286 8761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:08.091329 8761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:08.102882 8761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:08.104416 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:08.106009 8761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:08.121465 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:08.121922 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:08.121887 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:08.122666 8761 server_base.cc:1034] running on GCE node
I20260929 01:01:08.125039 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:08.127810 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:08.129503 8761 hybrid_clock.cc:648] HybridClock initialized: now 1790643668129389 us; error 71 us; skew 500 ppm
I20260929 01:01:08.130237 8761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:08 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:08.134654 8761 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:01:08.136870 8761 webserver.cc:460] Webserver started at http://127.3.21.254:35735/ using document root <none> and password file <none>
I20260929 01:01:08.137861 8761 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:08.138028 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:08.138415 8761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:08.143498 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "5b3d653230d9492fb72194b6d9a8e6bb"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-k0b7"
server_key: "f3b033b3c01608145a534212f5d3853f"
server_key_iv: "35938169aa462fd3bfc73311aad75894"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.144843 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "5b3d653230d9492fb72194b6d9a8e6bb"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-k0b7"
server_key: "f3b033b3c01608145a534212f5d3853f"
server_key_iv: "35938169aa462fd3bfc73311aad75894"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.152997 8761 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260929 01:01:08.158083 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:08.159797 8761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:08.159972 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "5b3d653230d9492fb72194b6d9a8e6bb"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-k0b7"
server_key: "f3b033b3c01608145a534212f5d3853f"
server_key_iv: "35938169aa462fd3bfc73311aad75894"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.160149 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:08.183233 8761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:08.188062 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:08.188381 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:08.208884 8761 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34001
I20260929 01:01:08.208918 8827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34001 every 8 connection(s)
I20260929 01:01:08.210287 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:08.212611 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8761
I20260929 01:01:08.212833 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:08.213618 3159 external_mini_cluster.cc:1428] Setting key d99a1999ea3c223e70796838dff9af15
I20260929 01:01:08.214828 8828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:01:08 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:08.230015 8828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb: Bootstrap starting.
I20260929 01:01:08.234828 8828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:08.236912 8828 log.cc:824] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:08.243170 8828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb: No bootstrap required, opened a new log
I20260929 01:01:08.244423 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:08.215883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.3.21.254:34001)
0929 01:01:08.216731 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:08.216749 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:08.216802 (+ 53us) server_negotiation.cc:407] Connection header received
0929 01:01:08.218827 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:08.218859 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:08.219503 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:08.220039 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:08.221393 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.223545 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.225058 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.226163 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.226379 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:08.231289 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:08.231301 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:08.231321 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:08.231377 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:08.236380 (+ 5003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:08.237399 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:08.237411 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:08.237420 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:08.237715 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:08.239624 (+ 1909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:08.239632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:08.239636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:08.240955 (+ 1319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:08.241581 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:08.242223 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 01:01:08.242896 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":326,"threads_started":1}
I20260929 01:01:08.252305 8828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34001 } }
I20260929 01:01:08.252852 8828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:08.252982 8828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3d653230d9492fb72194b6d9a8e6bb, State: Initialized, Role: FOLLOWER
I20260929 01:01:08.253659 8828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34001 } }
I20260929 01:01:08.253916 8828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:08.254071 8828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:08.254261 8828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:08.256681 8828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34001 } }
I20260929 01:01:08.257366 8828 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3d653230d9492fb72194b6d9a8e6bb; no voters:
I20260929 01:01:08.258010 8828 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:08.258193 8833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:08.259004 8833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [term 1 LEADER]: Becoming Leader. State: Replica: 5b3d653230d9492fb72194b6d9a8e6bb, State: Running, Role: LEADER
I20260929 01:01:08.259646 8833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34001 } }
I20260929 01:01:08.260476 8828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:08.263079 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3d653230d9492fb72194b6d9a8e6bb. Latest consensus state: current_term: 1 leader_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34001 } } }
I20260929 01:01:08.263058 8834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3d653230d9492fb72194b6d9a8e6bb" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34001 } } }
I20260929 01:01:08.263607 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:08.263687 8834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:08.264683 8841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:08.273267 8841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:08.284062 8841 catalog_manager.cc:1347] Generated new cluster ID: cb2d94141e014106935e07d0dac6d5e4
I20260929 01:01:08.284204 8841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:08.296514 8841 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:08.298768 8841 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:08.314270 8841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3d653230d9492fb72194b6d9a8e6bb: Generated new TSK 0
I20260929 01:01:08.315259 8841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:01:08.380844 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34001
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:08.742075 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:08.742409 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:08.742605 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:08.751132 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:08.751261 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:08.751382 8854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:08.762379 8854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:08.763841 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:08.765622 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:08.784370 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:08.784336 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:08.784309 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:08.785382 8854 server_base.cc:1034] running on GCE node
I20260929 01:01:08.786170 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:08.787364 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:08.788683 8854 hybrid_clock.cc:648] HybridClock initialized: now 1790643668788610 us; error 84 us; skew 500 ppm
I20260929 01:01:08.789110 8854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:08 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:08.793179 8854 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:01:08.795370 8854 webserver.cc:460] Webserver started at http://127.3.21.193:43321/ using document root <none> and password file <none>
I20260929 01:01:08.796198 8854 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:08.796365 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:08.796703 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:08.800081 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "054311a5e4ce4c43952f759bb7464327"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-k0b7"
server_key: "c9423188406e058d5af7c3ae6e6172f0"
server_key_iv: "e062a58b596bc9dfadc42bb5c949e4a9"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.800985 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "054311a5e4ce4c43952f759bb7464327"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-k0b7"
server_key: "c9423188406e058d5af7c3ae6e6172f0"
server_key_iv: "e062a58b596bc9dfadc42bb5c949e4a9"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.807364 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 01:01:08.811425 8869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:08.813328 8854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:01:08.813587 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "054311a5e4ce4c43952f759bb7464327"
format_stamp: "Formatted at 2026-09-29 01:01:08 on dist-test-slave-k0b7"
server_key: "c9423188406e058d5af7c3ae6e6172f0"
server_key_iv: "e062a58b596bc9dfadc42bb5c949e4a9"
server_key_version: "encryptionkey@0"
I20260929 01:01:08.813838 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:08.837718 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:08.843916 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:08.844276 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:08.845448 8854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:08.847222 8854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:08.847347 8854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:08.847502 8854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:08.847576 8854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:08.890499 8854 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:37997
I20260929 01:01:08.890625 8981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:37997 every 8 connection(s)
I20260929 01:01:08.891968 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:08.895954 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8854
I20260929 01:01:08.896214 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:08.897011 3159 external_mini_cluster.cc:1428] Setting key e3681ba26a442fa770dde984444b58da
Sep 29 01:01:08 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:01:08.921233 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:08.894944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:41313 (local address 127.3.21.254:34001)
0929 01:01:08.895648 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:08.895657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:08.897243 (+ 1586us) server_negotiation.cc:407] Connection header received
0929 01:01:08.900081 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:08.900087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:08.900198 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:08.900452 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:08.902324 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.903533 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.906533 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.907403 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.907604 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:08.912900 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:08.912907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:08.912911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:08.912952 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:08.916291 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:08.917606 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:08.917767 (+ 161us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:08.917776 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:08.918102 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:08.918971 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:08.918978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:08.918981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:08.919330 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:08.919479 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:08.920531 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0929 01:01:08.920803 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":403,"threads_started":1}
I20260929 01:01:08.922564 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:08.895875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34001 (local address 127.3.21.193:41313)
0929 01:01:08.897004 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:08.897058 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:08.899571 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:08.900744 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:08.900754 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:08.901445 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:08.902093 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:08.902113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.903782 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.903788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:08.906158 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:08.906183 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:08.907858 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:08.907868 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:08.908260 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:08.909178 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:08.909217 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:08.912550 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:08.916631 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:08.916652 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:08.916675 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:08.917379 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:08.918353 (+ 974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:08.918360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:08.918365 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:08.918807 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:08.919652 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:08.919660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:08.920089 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 01:01:08.920412 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 01:01:08.921071 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":564,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:01:08.924630 8982 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34001
I20260929 01:01:08.924994 8982 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:08.925853 8982 heartbeater.cc:511] Master 127.3.21.254:34001 requested a full tablet report, sending...
I20260929 01:01:08.928483 8793 ts_manager.cc:194] Registered new tserver with Master: 054311a5e4ce4c43952f759bb7464327 (127.3.21.193:37997)
I20260929 01:01:08.930248 8793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:41313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:01:08.955508 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34001
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:09.323678 8988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:09.324009 8988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:09.324177 8988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:09.332007 8988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:09.332141 8988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:09.332317 8988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:09.344339 8988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:09.345815 8988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:09.347340 8988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:09.362844 8996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:09.363013 8994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:09.363711 8993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:09.365034 8988 server_base.cc:1034] running on GCE node
I20260929 01:01:09.366034 8988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:09.367800 8988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:09.371279 8988 hybrid_clock.cc:648] HybridClock initialized: now 1790643669371292 us; error 140 us; skew 500 ppm
I20260929 01:01:09.371824 8988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:09 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:09.376001 8988 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:01:09.378073 8988 webserver.cc:460] Webserver started at http://127.3.21.194:45257/ using document root <none> and password file <none>
I20260929 01:01:09.378989 8988 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:09.379158 8988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:09.379499 8988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:09.382910 8988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "a1be491f77fc468a91c7f75fc1d56b50"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-k0b7"
server_key: "072b28e9fbbfa93f475e2679fc674ad7"
server_key_iv: "c8d33f110f717d233e30be0ef342d5f8"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.383837 8988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "a1be491f77fc468a91c7f75fc1d56b50"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-k0b7"
server_key: "072b28e9fbbfa93f475e2679fc674ad7"
server_key_iv: "c8d33f110f717d233e30be0ef342d5f8"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.390923 8988 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 01:01:09.396051 9003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:09.398128 8988 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:01:09.398432 8988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "a1be491f77fc468a91c7f75fc1d56b50"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-k0b7"
server_key: "072b28e9fbbfa93f475e2679fc674ad7"
server_key_iv: "c8d33f110f717d233e30be0ef342d5f8"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.398794 8988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:09.415797 8988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:09.421465 8988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:09.421823 8988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:09.423084 8988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:09.424595 8988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:09.424695 8988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:09.424834 8988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:09.424932 8988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:09.466444 8988 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:37867
I20260929 01:01:09.466559 9115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:37867 every 8 connection(s)
I20260929 01:01:09.468164 8988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:09.472414 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 8988
I20260929 01:01:09.472651 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:09.473315 3159 external_mini_cluster.cc:1428] Setting key 2d0102c3d19583156d740c53d64d60fd
Sep 29 01:01:09 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:09.493964 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:09.471181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:59917 (local address 127.3.21.254:34001)
0929 01:01:09.472129 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:09.472134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:09.473164 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 01:01:09.475583 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:09.475588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:09.475674 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:09.475796 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:09.478028 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:09.479619 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:09.481527 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:09.481987 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:09.482138 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:09.486680 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:09.486686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:09.486689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:09.486727 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:09.490111 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:09.491219 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:09.491225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:09.491234 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:09.491379 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:09.491913 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:09.491919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:09.491922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:09.492227 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:09.492407 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:09.493241 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 01:01:09.493536 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":721,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:01:09.495301 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:09.471628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34001 (local address 127.3.21.194:59917)
0929 01:01:09.472957 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:09.473018 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:09.475226 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:09.476107 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:09.476118 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:09.476751 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:09.477505 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:09.477528 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:09.479906 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:09.479911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:09.481321 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:09.481338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:09.482273 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:09.482282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:09.482622 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:09.483454 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:09.483480 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:09.486281 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:09.490360 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:09.490373 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:09.490395 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:09.491048 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:09.491532 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:09.491538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:09.491542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:09.491774 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:09.492381 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:09.492390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:09.492814 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 01:01:09.493126 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 01:01:09.493780 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":508,"threads_started":1}
I20260929 01:01:09.497431 9116 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34001
I20260929 01:01:09.497742 9116 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:01:09.498696 9116 heartbeater.cc:511] Master 127.3.21.254:34001 requested a full tablet report, sending...
I20260929 01:01:09.500564 8793 ts_manager.cc:194] Registered new tserver with Master: a1be491f77fc468a91c7f75fc1d56b50 (127.3.21.194:37867)
I20260929 01:01:09.501973 8793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:59917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:01:09.531355 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34001
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:09.904362 9122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:09.904906 9122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:09.905098 9122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:09.914801 9122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:09.914948 9122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:09.915139 9122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:09.926209 9122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:09.927677 9122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:09.929008 9122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:01:09.934710 8982 heartbeater.cc:503] Master 127.3.21.254:34001 was elected leader, sending a full tablet report...
W20260929 01:01:09.945322 9128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:09.945130 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:09.945204 9127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:09.946328 9122 server_base.cc:1034] running on GCE node
I20260929 01:01:09.947168 9122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:09.948330 9122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:09.949620 9122 hybrid_clock.cc:648] HybridClock initialized: now 1790643669949573 us; error 46 us; skew 500 ppm
I20260929 01:01:09.950040 9122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:09 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:09.954463 9122 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:01:09.956254 9122 webserver.cc:460] Webserver started at http://127.3.21.195:32805/ using document root <none> and password file <none>
I20260929 01:01:09.957171 9122 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:09.957329 9122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:09.957659 9122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:09.960983 9122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "f0b1b52545ad401aad4eb37958bca576"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-k0b7"
server_key: "608249694d13bfca783a1378f8a0732d"
server_key_iv: "da6e639c5928615b54a8988b9cdb3b95"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.961860 9122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "f0b1b52545ad401aad4eb37958bca576"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-k0b7"
server_key: "608249694d13bfca783a1378f8a0732d"
server_key_iv: "da6e639c5928615b54a8988b9cdb3b95"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.968271 9122 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:01:09.972846 9137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:09.974736 9122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:09.975044 9122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "f0b1b52545ad401aad4eb37958bca576"
format_stamp: "Formatted at 2026-09-29 01:01:09 on dist-test-slave-k0b7"
server_key: "608249694d13bfca783a1378f8a0732d"
server_key_iv: "da6e639c5928615b54a8988b9cdb3b95"
server_key_version: "encryptionkey@0"
I20260929 01:01:09.975405 9122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:10.022616 9122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:10.030628 9122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:10.031018 9122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:10.032192 9122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:10.033756 9122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:10.033882 9122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:10.034044 9122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:10.034118 9122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:10.081578 9122 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:38713
I20260929 01:01:10.081831 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:38713 every 8 connection(s)
I20260929 01:01:10.083186 9122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:10.087303 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9122
I20260929 01:01:10.087509 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:01:10.088312 3159 external_mini_cluster.cc:1428] Setting key 4aa86343673995e052103952d28a5907
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:10.116247 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.088767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:46955 (local address 127.3.21.254:34001)
0929 01:01:10.089385 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.089392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.091579 (+ 2187us) server_negotiation.cc:407] Connection header received
0929 01:01:10.096296 (+ 4717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.096304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.096406 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.096551 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.099141 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.100180 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.102733 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.103199 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.103504 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.108104 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.108109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.108112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.108148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.111895 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.112849 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.112854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.112856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.113008 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.113846 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.113852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.113855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.114621 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.114776 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.115684 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.115905 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":363,"threads_started":1}
I20260929 01:01:10.117391 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.089428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34001 (local address 127.3.21.195:46955)
0929 01:01:10.091381 (+ 1953us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:10.091438 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:10.095434 (+ 3996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:10.096889 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:10.096900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:10.097605 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:10.098756 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:10.098796 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.100556 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.100562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:10.102223 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:10.102243 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.103597 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.103603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:10.103968 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.104987 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:10.105015 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:10.107794 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:10.112120 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:10.112130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:10.112148 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:10.112691 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:10.113149 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:10.113153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:10.113158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:10.113577 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:10.114808 (+ 1231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:10.114816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:10.115207 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 01:01:10.115478 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 01:01:10.116033 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":1187,"threads_started":1}
I20260929 01:01:10.119711 9250 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34001
I20260929 01:01:10.120188 9250 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:10.121086 9250 heartbeater.cc:511] Master 127.3.21.254:34001 requested a full tablet report, sending...
I20260929 01:01:10.123549 8793 ts_manager.cc:194] Registered new tserver with Master: f0b1b52545ad401aad4eb37958bca576 (127.3.21.195:38713)
I20260929 01:01:10.124462 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:01:10.124670 8793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:46955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/test-admin.keytab.
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.163200 3159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 01:01:10.169510 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:10.166563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.3.21.193:37997)
0929 01:01:10.167278 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.167286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.167314 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:01:10.167380 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.167386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.167605 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.167814 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.169039 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48840: BlockingRecv error: recv got EOF from 127.0.0.1:48840 (error 108)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":354,"threads_started":1}
W20260929 01:01:10.169761 3159 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.171702 3159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.173182 3159 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:01:10.195283 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.175283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.3.21.193:37997)
0929 01:01:10.175861 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.175869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.176067 (+ 198us) server_negotiation.cc:407] Connection header received
0929 01:01:10.176284 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.176289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.176364 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.176563 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.177709 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.178944 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.180582 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.181279 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.181393 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.186846 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.186866 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.187038 (+ 172us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.187092 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.191456 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.192503 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.192512 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.192515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.192667 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.193202 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.193207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.193210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.193706 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.194102 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.194581 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.194960 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
W20260929 01:01:10.204236 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:10.201083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.3.21.193:37997)
0929 01:01:10.201564 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.201576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.202154 (+ 578us) server_negotiation.cc:407] Connection header received
0929 01:01:10.202423 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.202436 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.202699 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.202879 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.203914 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48862: BlockingRecv error: recv got EOF from 127.0.0.1:48862 (error 108)
Metrics: {"server-negotiator.queue_time_us":258}
W20260929 01:01:10.204751 3159 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.206873 3159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.208783 3159 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:01:10.226177 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.210393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.3.21.193:37997)
0929 01:01:10.210709 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.210718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.211052 (+ 334us) server_negotiation.cc:407] Connection header received
0929 01:01:10.211278 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.211282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.211345 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.211452 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.212458 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.213171 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.214451 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.214975 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.215111 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.220031 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.220040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.220044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.220093 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.223318 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.224108 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.224118 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.224122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.224269 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.224686 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.224692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.224695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.225044 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.225322 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.225614 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.225853 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
W20260929 01:01:10.231622 9258 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:10.229323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.3.21.193:37997)
0929 01:01:10.229664 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.229673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.229932 (+ 259us) server_negotiation.cc:407] Connection header received
0929 01:01:10.230152 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.230157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.230243 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.230361 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.231330 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48882: BlockingRecv error: recv got EOF from 127.0.0.1:48882 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
W20260929 01:01:10.231972 3159 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643611838815-3159-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.233858 3159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:10.235989 3159 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:01:10.255677 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.237669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.3.21.193:37997)
0929 01:01:10.237955 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.237964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.238597 (+ 633us) server_negotiation.cc:407] Connection header received
0929 01:01:10.238666 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.238670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.238736 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.238835 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.240185 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.241029 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.242627 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.243079 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.243187 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.248258 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.248264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.248267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.248310 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.251860 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.252854 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.252859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.252863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.253051 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.253666 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.253670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.253672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.254049 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.254304 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.255060 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.255311 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 01:01:10.257835 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8854
I20260929 01:01:10.275993 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.258681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.3.21.193:37997)
0929 01:01:10.259141 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.259146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.259512 (+ 366us) server_negotiation.cc:407] Connection header received
0929 01:01:10.259580 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.259584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.259659 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.259749 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.261118 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.262303 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.263854 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.264301 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.264419 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.267809 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.267815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.267819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.267867 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.272089 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.273061 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.273066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.273069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.273235 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.273815 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.273819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.273822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.274217 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.274475 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.275065 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.275646 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260929 01:01:10.276587 8977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:10.378966 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8854
I20260929 01:01:10.398811 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8988
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:01:10.418682 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.399948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.3.21.194:37867)
0929 01:01:10.400724 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.400730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.400757 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:01:10.400830 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.400839 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.401091 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.401235 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.402473 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.403889 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.405380 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.405960 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.406072 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.411132 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.411150 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.411176 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.411230 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.414874 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.415857 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.415866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.415868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.416025 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.416704 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.416722 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.416725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.417149 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.417570 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.417853 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.418096 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":497,"threads_started":1}
I20260929 01:01:10.420213 9111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:10.520116 9116 heartbeater.cc:503] Master 127.3.21.254:34001 was elected leader, sending a full tablet report...
I20260929 01:01:10.521575 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8988
W20260929 01:01:10.539304 8780 connection.cc:535] server connection from 127.3.21.194:59917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:01:10.539705 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9122
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[8748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:01:10.562299 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:10.541214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.3.21.195:38713)
0929 01:01:10.542423 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:10.542433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:10.542468 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:10.542632 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:10.542645 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:10.542934 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:10.543288 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:10.544954 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.546908 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.548547 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:10.549394 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:10.549587 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:10.554734 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:10.554760 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:10.554782 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:10.554853 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:10.558402 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.559407 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.559416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.559418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.559733 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:10.560255 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:10.560259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:10.560261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:10.560824 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:10.561092 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:10.561620 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 01:01:10.561964 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":595,"threads_started":1}
I20260929 01:01:10.564996 9245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:10.662235 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9122
I20260929 01:01:10.679597 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 8761
I20260929 01:01:10.681195 8823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:10.794700 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 8761
2026-09-29T01:01:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4513 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[9278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[9278](info): set up 2 sockets
Sep 29 01:01:10 dist-test-slave-k0b7 krb5kdc[9278](info): commencing operation
krb5kdc: starting...
W20260929 01:01:11.987639 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:01:12 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab.
Entry for principal oryx/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab.
I20260929 01:01:12.249248 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36493
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab
--principal=oryx/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:12.632942 9291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:12.633364 9291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:12.633534 9291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:12.641906 9291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:12.642066 9291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:12.642117 9291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:12.642170 9291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:12.642210 9291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:12.655297 9291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab
--principal=oryx/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:12.656984 9291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:12.658349 9291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:12.672914 9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:12.673028 9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:12.673213 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:12.674337 9291 server_base.cc:1034] running on GCE node
I20260929 01:01:12.675333 9291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:12.677021 9291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:12.678396 9291 hybrid_clock.cc:648] HybridClock initialized: now 1790643672678359 us; error 195 us; skew 500 ppm
I20260929 01:01:12.679203 9291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:12 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:12.683298 9291 init.cc:377] Logged in from keytab as oryx/127.3.21.254@KRBTEST.COM (short username oryx)
I20260929 01:01:12.685245 9291 webserver.cc:460] Webserver started at http://127.3.21.254:44801/ using document root <none> and password file <none>
I20260929 01:01:12.686091 9291 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:12.686251 9291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:12.686655 9291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:12.690119 9291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "59123c9daf244e67983512edc364a35a"
format_stamp: "Formatted at 2026-09-29 01:01:12 on dist-test-slave-k0b7"
server_key: "f08f4e2f3ed3cf759b48e9e3a14a4531"
server_key_iv: "5944f7ea3efceee475d02278492f7573"
server_key_version: "encryptionkey@0"
I20260929 01:01:12.691054 9291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "59123c9daf244e67983512edc364a35a"
format_stamp: "Formatted at 2026-09-29 01:01:12 on dist-test-slave-k0b7"
server_key: "f08f4e2f3ed3cf759b48e9e3a14a4531"
server_key_iv: "5944f7ea3efceee475d02278492f7573"
server_key_version: "encryptionkey@0"
I20260929 01:01:12.697566 9291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260929 01:01:12.702384 9306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:12.704689 9291 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:12.705039 9291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "59123c9daf244e67983512edc364a35a"
format_stamp: "Formatted at 2026-09-29 01:01:12 on dist-test-slave-k0b7"
server_key: "f08f4e2f3ed3cf759b48e9e3a14a4531"
server_key_iv: "5944f7ea3efceee475d02278492f7573"
server_key_version: "encryptionkey@0"
I20260929 01:01:12.705318 9291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:12.726104 9291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:12.731645 9291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:12.732019 9291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:12.754076 9291 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36493
I20260929 01:01:12.754094 9357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36493 every 8 connection(s)
I20260929 01:01:12.755653 9291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:12.756973 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9291
I20260929 01:01:12.757184 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:12.757833 3159 external_mini_cluster.cc:1428] Setting key daa5640514f9e55fb162c3c98b606f1b
I20260929 01:01:12.761523 9358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:01:12 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:12.777504 9358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Bootstrap starting.
I20260929 01:01:12.782449 9358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:12.783970 9358 log.cc:824] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:12.787241 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:12.760379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.3.21.254:36493)
0929 01:01:12.761230 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:12.761243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:12.761295 (+ 52us) server_negotiation.cc:407] Connection header received
0929 01:01:12.763622 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:12.763657 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:12.764273 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:12.764848 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:12.767367 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:12.769767 (+ 2400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:12.772048 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:12.772911 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:12.773137 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:12.777483 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:12.777496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:12.777515 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:12.777566 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:12.781950 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:12.783103 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:12.783111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:12.783119 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:12.783305 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:12.783738 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:12.783748 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:12.783750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:12.784446 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:12.784801 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:12.785278 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 01:01:12.785807 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":369,"threads_started":1}
I20260929 01:01:12.787983 9358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: No bootstrap required, opened a new log
I20260929 01:01:12.796461 9358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } }
I20260929 01:01:12.796876 9358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:12.797009 9358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59123c9daf244e67983512edc364a35a, State: Initialized, Role: FOLLOWER
I20260929 01:01:12.797721 9358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } }
I20260929 01:01:12.798064 9358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:12.798216 9358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:12.798394 9358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:12.800725 9358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } }
I20260929 01:01:12.801303 9358 leader_election.cc:302] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59123c9daf244e67983512edc364a35a; no voters:
I20260929 01:01:12.801949 9358 leader_election.cc:288] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:12.802203 9363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:12.803068 9363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 LEADER]: Becoming Leader. State: Replica: 59123c9daf244e67983512edc364a35a, State: Running, Role: LEADER
I20260929 01:01:12.803776 9363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } }
I20260929 01:01:12.804440 9358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:12.806984 9365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59123c9daf244e67983512edc364a35a. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } }
I20260929 01:01:12.807390 9365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:12.808028 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } }
I20260929 01:01:12.809171 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:12.810034 9372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:12.816184 9372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:12.830914 9372 catalog_manager.cc:1347] Generated new cluster ID: f310d5b134134361914cb13b4b9ab2cf
I20260929 01:01:12.831074 9372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:12.876225 9372 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:12.877981 9372 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:12.891033 9372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Generated new TSK 0
I20260929 01:01:12.892119 9372 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab.
Entry for principal oryx/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab.
I20260929 01:01:12.964381 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36493
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:13.346371 9385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:13.346733 9385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:13.346907 9385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:13.354616 9385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:13.354748 9385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:13.354964 9385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:13.366444 9385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:13.368088 9385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:13.369835 9385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:13.387650 9391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:13.387598 9390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:13.387821 9393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:13.388916 9385 server_base.cc:1034] running on GCE node
I20260929 01:01:13.389696 9385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:13.391007 9385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:13.392233 9385 hybrid_clock.cc:648] HybridClock initialized: now 1790643673392147 us; error 68 us; skew 500 ppm
I20260929 01:01:13.392661 9385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:13 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643673, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:13.407727 9385 init.cc:377] Logged in from keytab as oryx/127.3.21.193@KRBTEST.COM (short username oryx)
I20260929 01:01:13.409730 9385 webserver.cc:460] Webserver started at http://127.3.21.193:40525/ using document root <none> and password file <none>
I20260929 01:01:13.410701 9385 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:13.410874 9385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:13.411223 9385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:13.414387 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "b775d706489f4d0ea3084db13ee49cb9"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "f143b7bf3446a55afa96571c3d3fdfbf"
server_key_iv: "723c2a51f40838ba610b262fc18b6f29"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.415341 9385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "b775d706489f4d0ea3084db13ee49cb9"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "f143b7bf3446a55afa96571c3d3fdfbf"
server_key_iv: "723c2a51f40838ba610b262fc18b6f29"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.421164 9385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:01:13.425294 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:13.427044 9385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:13.427258 9385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "b775d706489f4d0ea3084db13ee49cb9"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "f143b7bf3446a55afa96571c3d3fdfbf"
server_key_iv: "723c2a51f40838ba610b262fc18b6f29"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.427490 9385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:13.458817 9385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:13.464255 9385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:13.464628 9385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:13.465791 9385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:13.467293 9385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:13.467428 9385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:13.467600 9385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:13.467684 9385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:13.506176 9385 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:33573
I20260929 01:01:13.506206 9512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:33573 every 8 connection(s)
I20260929 01:01:13.507665 9385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:13.510754 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9385
I20260929 01:01:13.511013 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:13.511895 3159 external_mini_cluster.cc:1428] Setting key db699d951e6c8f70d0bc7d361715f595
Sep 29 01:01:13 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643673, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:13.532749 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:13.510649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:59465 (local address 127.3.21.254:36493)
0929 01:01:13.511168 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:13.511174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:13.512600 (+ 1426us) server_negotiation.cc:407] Connection header received
0929 01:01:13.514982 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:13.514988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:13.515086 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:13.515229 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:13.517128 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:13.518461 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:13.520507 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:13.521076 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:13.521193 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:13.526398 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:13.526404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:13.526407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:13.526443 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:13.529244 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:13.530103 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:13.530109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:13.530112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:13.530230 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:13.530748 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:13.530756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:13.530760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:13.531050 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:13.531213 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:13.532084 (+ 871us) server_negotiation.cc:299] Negotiation successful
0929 01:01:13.532330 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":290,"threads_started":1}
I20260929 01:01:13.533783 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:13.511354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.193:59465)
0929 01:01:13.512374 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:13.512443 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:13.514655 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:13.515513 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:13.515524 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:13.516222 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:13.516915 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:13.516937 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:13.518691 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:13.518696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:13.520190 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:13.520212 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:13.521508 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:13.521519 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:13.521923 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:13.522884 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:13.522925 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:13.526106 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:13.529429 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:13.529439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:13.529466 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:13.529956 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:13.530358 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:13.530364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:13.530367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:13.530634 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:13.531191 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:13.531199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:13.531612 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 01:01:13.531969 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 01:01:13.532569 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":435,"threads_started":1}
I20260929 01:01:13.535785 9513 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36493
I20260929 01:01:13.536187 9513 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:01:13.537180 9513 heartbeater.cc:511] Master 127.3.21.254:36493 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.194@KRBTEST.COM" created.
I20260929 01:01:13.540308 9323 ts_manager.cc:194] Registered new tserver with Master: b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:13.542155 9323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:59465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab.
Entry for principal oryx/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab.
I20260929 01:01:13.574892 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36493
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:13.943488 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:13.943801 9519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:13.943902 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:13.952353 9519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:13.952512 9519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:13.952634 9519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:13.963868 9519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:13.965443 9519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:13.966861 9519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:13.984643 9524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:13.984511 9525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:13.984437 9527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:13.985473 9519 server_base.cc:1034] running on GCE node
I20260929 01:01:13.986218 9519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:13.987614 9519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:13.988886 9519 hybrid_clock.cc:648] HybridClock initialized: now 1790643673988883 us; error 90 us; skew 500 ppm
I20260929 01:01:13.989241 9519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:13 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643673, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:13.992754 9519 init.cc:377] Logged in from keytab as oryx/127.3.21.194@KRBTEST.COM (short username oryx)
I20260929 01:01:13.994396 9519 webserver.cc:460] Webserver started at http://127.3.21.194:34829/ using document root <none> and password file <none>
I20260929 01:01:13.995215 9519 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:13.995328 9519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:13.995577 9519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:13.998764 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "93edca9da1bcf6c6d0c38f711bd01228"
server_key_iv: "33aa777197b0e8802db4dc870c82a1d8"
server_key_version: "encryptionkey@0"
I20260929 01:01:13.999569 9519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "93edca9da1bcf6c6d0c38f711bd01228"
server_key_iv: "33aa777197b0e8802db4dc870c82a1d8"
server_key_version: "encryptionkey@0"
I20260929 01:01:14.008438 9519 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 01:01:14.014060 9534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:14.016029 9519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:01:14.016279 9519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "93edca9da1bcf6c6d0c38f711bd01228"
server_key_iv: "33aa777197b0e8802db4dc870c82a1d8"
server_key_version: "encryptionkey@0"
I20260929 01:01:14.016567 9519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:14.035694 9519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:14.040726 9519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:14.041069 9519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:14.042186 9519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:14.044198 9519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:14.044327 9519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:14.044497 9519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:14.044575 9519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:14.082626 9519 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:44107
I20260929 01:01:14.082644 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:44107 every 8 connection(s)
I20260929 01:01:14.084118 9519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:14.090068 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9519
I20260929 01:01:14.090262 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:14.090888 3159 external_mini_cluster.cc:1428] Setting key b9c7e0b78b96dcecfae9a55b31fa3802
Sep 29 01:01:14 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643673, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:14.110152 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:14.087134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:47229 (local address 127.3.21.254:36493)
0929 01:01:14.087646 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:14.087653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:14.088804 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 01:01:14.091450 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:14.091454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:14.091531 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:14.091646 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:14.093598 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.094862 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.096700 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.097457 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.097612 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:14.102018 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:14.102024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:14.102027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:14.102064 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:14.105174 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:14.106792 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:14.106801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:14.106805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:14.106991 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:14.107796 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:14.107802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:14.107805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:14.108184 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:14.108376 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:14.109439 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0929 01:01:14.109698 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":305,"threads_started":1}
I20260929 01:01:14.111289 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:14.087740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:47229)
0929 01:01:14.088593 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:14.088637 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:14.091030 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:14.091996 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:14.092007 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:14.092579 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:14.093366 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:14.093394 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.095105 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.095110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:14.096517 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:14.096532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.097704 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.097709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:14.097953 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:14.098853 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:14.098878 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:14.101694 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:14.105623 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:14.105657 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:14.105682 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:14.106599 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:14.107357 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:14.107365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:14.107372 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:14.107652 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:14.108374 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:14.108385 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:14.108993 (+ 608us) client_negotiation.cc:769] Sending connection context
0929 01:01:14.109311 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 01:01:14.109930 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":317,"threads_started":1}
WARNING: no policy specified for oryx/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:01:14.113574 9647 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36493
I20260929 01:01:14.113981 9647 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:14.114975 9647 heartbeater.cc:511] Master 127.3.21.254:36493 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.195@KRBTEST.COM" created.
I20260929 01:01:14.117561 9323 ts_manager.cc:194] Registered new tserver with Master: 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107)
I20260929 01:01:14.118959 9323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:47229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab.
Entry for principal oryx/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab.
I20260929 01:01:14.151940 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36493
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:14.516173 9653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:14.516547 9653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:14.516714 9653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:14.524345 9653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:14.524518 9653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:14.524688 9653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:14.534940 9653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:14.536388 9653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:14.537765 9653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:01:14.546000 9513 heartbeater.cc:503] Master 127.3.21.254:36493 was elected leader, sending a full tablet report...
W20260929 01:01:14.552917 9658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:14.552908 9659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:14.553056 9661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:14.554008 9653 server_base.cc:1034] running on GCE node
I20260929 01:01:14.554955 9653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:14.556003 9653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:14.557253 9653 hybrid_clock.cc:648] HybridClock initialized: now 1790643674557206 us; error 44 us; skew 500 ppm
I20260929 01:01:14.557690 9653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:14 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643674, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:14.561473 9653 init.cc:377] Logged in from keytab as oryx/127.3.21.195@KRBTEST.COM (short username oryx)
I20260929 01:01:14.563408 9653 webserver.cc:460] Webserver started at http://127.3.21.195:41983/ using document root <none> and password file <none>
I20260929 01:01:14.564208 9653 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:14.564368 9653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:14.564699 9653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:14.568001 9653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "315db3dbd8e94f5db25b298148d6ce54"
format_stamp: "Formatted at 2026-09-29 01:01:14 on dist-test-slave-k0b7"
server_key: "c4e16659c36d5b0207ec01d5efc96f32"
server_key_iv: "766ff9402925236671f7997e39b0631b"
server_key_version: "encryptionkey@0"
I20260929 01:01:14.568898 9653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "315db3dbd8e94f5db25b298148d6ce54"
format_stamp: "Formatted at 2026-09-29 01:01:14 on dist-test-slave-k0b7"
server_key: "c4e16659c36d5b0207ec01d5efc96f32"
server_key_iv: "766ff9402925236671f7997e39b0631b"
server_key_version: "encryptionkey@0"
I20260929 01:01:14.575246 9653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 01:01:14.579588 9668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:14.581138 9653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:01:14.581355 9653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "315db3dbd8e94f5db25b298148d6ce54"
format_stamp: "Formatted at 2026-09-29 01:01:14 on dist-test-slave-k0b7"
server_key: "c4e16659c36d5b0207ec01d5efc96f32"
server_key_iv: "766ff9402925236671f7997e39b0631b"
server_key_version: "encryptionkey@0"
I20260929 01:01:14.581594 9653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:14.599985 9653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:14.605451 9653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:14.605780 9653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:14.607137 9653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:14.608981 9653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:14.609097 9653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:14.609244 9653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:14.609325 9653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:14.647871 9653 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37327
I20260929 01:01:14.648044 9780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37327 every 8 connection(s)
I20260929 01:01:14.649242 9653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:14.657290 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9653
I20260929 01:01:14.657487 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:01:14.658054 3159 external_mini_cluster.cc:1428] Setting key eecb4c73e94771282dc62bffc5e34518
Sep 29 01:01:14 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643674, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:14.673297 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:14.652525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:33541 (local address 127.3.21.254:36493)
0929 01:01:14.653040 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:14.653045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:14.654127 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 01:01:14.656149 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:14.656154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:14.656251 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:14.656371 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:14.658973 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.660143 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.662149 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.662681 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.662863 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:14.666991 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:14.666997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:14.667001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:14.667036 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:14.669721 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:14.670703 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:14.670707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:14.670710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:14.670840 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:14.671289 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:14.671293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:14.671295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:14.671581 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:14.671699 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:14.672635 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 01:01:14.672884 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":277,"threads_started":1}
I20260929 01:01:14.674822 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:14.653133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:33541)
0929 01:01:14.653937 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:14.653983 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:14.655859 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:14.656797 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:14.656816 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:14.657685 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:14.658760 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:14.658784 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.660373 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.660378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:14.661981 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:14.661995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:14.662990 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:14.662999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:14.663307 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:14.664150 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:14.664178 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:14.666716 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:14.669906 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:14.669916 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:14.669937 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:14.670537 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:14.670960 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:14.670964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:14.670966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:14.671157 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:14.671700 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:14.671707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:14.672078 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 01:01:14.672545 (+ 467us) client_negotiation.cc:240] Negotiation successful
0929 01:01:14.673110 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":311,"threads_started":1}
I20260929 01:01:14.676698 9781 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36493
I20260929 01:01:14.677062 9781 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:14.677994 9781 heartbeater.cc:511] Master 127.3.21.254:36493 requested a full tablet report, sending...
I20260929 01:01:14.679738 9323 ts_manager.cc:194] Registered new tserver with Master: 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195:37327)
I20260929 01:01:14.680709 9323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:33541
I20260929 01:01:14.687971 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.21.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.253.keytab.
Entry for principal oryx/127.3.21.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.253.keytab.
I20260929 01:01:14.745353 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:33935
--webserver_interface=127.3.21.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.253.keytab
--principal=oryx/127.3.21.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36493,127.3.21.253:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:15.101260 9787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:15.101596 9787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:15.101689 9787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:15.109536 9787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:15.109674 9787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:15.109720 9787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:15.109825 9787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:15.109889 9787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:15.121057 9787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36493,127.3.21.253:33935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.253.keytab
--principal=oryx/127.3.21.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.21.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
W20260929 01:01:15.121495 9787 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.21.254:36493,127.3.21.253:33935'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 01:01:15.122411 9647 heartbeater.cc:503] Master 127.3.21.254:36493 was elected leader, sending a full tablet report...
I20260929 01:01:15.122723 9787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:15.124266 9787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:15.138724 9792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.138701 9795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.139007 9793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:15.139747 9787 server_base.cc:1034] running on GCE node
I20260929 01:01:15.140906 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:15.143330 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:15.144681 9787 hybrid_clock.cc:648] HybridClock initialized: now 1790643675144612 us; error 58 us; skew 500 ppm
I20260929 01:01:15.145283 9787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:15.149420 9787 init.cc:377] Logged in from keytab as oryx/127.3.21.253@KRBTEST.COM (short username oryx)
I20260929 01:01:15.151271 9787 webserver.cc:460] Webserver started at http://127.3.21.253:44789/ using document root <none> and password file <none>
I20260929 01:01:15.152146 9787 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:15.152287 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:15.152613 9787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:15.155864 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/instance:
uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "692fe48e139c55675689fe57256de3b6"
server_key_iv: "9aeefac840623a64ef0db5fc90efe092"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.156735 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal/instance:
uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "692fe48e139c55675689fe57256de3b6"
server_key_iv: "9aeefac840623a64ef0db5fc90efe092"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.162635 9787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 01:01:15.166666 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:15.168519 9787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:01:15.168746 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "692fe48e139c55675689fe57256de3b6"
server_key_iv: "9aeefac840623a64ef0db5fc90efe092"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.169011 9787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:15.197459 9787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:15.202702 9787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:15.202993 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:15.223423 9787 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.253:33935
I20260929 01:01:15.223482 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.253:33935 every 8 connection(s)
I20260929 01:01:15.224922 9787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
I20260929 01:01:15.228946 9854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:15.229373 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9787
I20260929 01:01:15.239288 9854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.253@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:15.256126 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.231711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.3.21.253:33935)
0929 01:01:15.232619 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.232631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.232672 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:01:15.234165 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.234189 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.234732 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.235110 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.237023 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.239088 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.240570 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.241265 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.241499 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.246649 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.246659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.246676 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.246722 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.251038 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.252099 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.252115 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.252126 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.252323 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.252767 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.252773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.252776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.253372 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.253711 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.254254 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.254809 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":399,"threads_started":1}
I20260929 01:01:15.260459 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.242083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.3.21.254:36493)
0929 01:01:15.242627 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.242631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.243123 (+ 492us) server_negotiation.cc:407] Connection header received
0929 01:01:15.243295 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.243301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.243391 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.243535 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.244710 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.245541 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.247575 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.248095 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.248200 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.252894 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.252899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.252902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.252950 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.255814 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.257978 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.257983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.257985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.258128 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.258644 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.258649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.258651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.259134 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.259319 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.259822 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.260069 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":309,"threads_started":1}
I20260929 01:01:15.260524 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.242052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58518)
0929 01:01:15.242967 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.243010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.243166 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.243615 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.243620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.244262 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:15.244539 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.244548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.245734 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.245739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.247305 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.247321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.248436 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.248445 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.248677 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.249421 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:15.249446 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:15.252603 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:15.255979 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.255989 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.255996 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.257794 (+ 1798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.258258 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.258263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.258267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.258450 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.259306 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:15.259313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:15.259492 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.259691 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.260059 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":513,"threads_started":1}
I20260929 01:01:15.262239 9854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.253@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
I20260929 01:01:15.277259 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.263169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.3.21.253:33935)
0929 01:01:15.263501 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.263507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.263617 (+ 110us) server_negotiation.cc:407] Connection header received
0929 01:01:15.263854 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.263858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.263941 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.264055 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.264776 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.265761 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.267244 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.267708 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.267863 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.271518 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.271524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.271527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.271560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.274575 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.275463 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.275467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.275470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.275586 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.275956 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.275959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.275962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.276246 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.276422 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.276788 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.277006 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":95}
I20260929 01:01:15.277663 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.263169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47298)
0929 01:01:15.263514 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.263535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.263687 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.264070 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.264074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.264489 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:15.264630 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.264638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.266017 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.266021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.267091 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.267104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.267900 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.267906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.268113 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.268843 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:15.268862 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:15.271242 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:15.274756 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.274760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.274763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.275169 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.275711 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.275714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.275717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.275856 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.276404 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:15.276409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:15.276568 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.276701 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.277191 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":174}
I20260929 01:01:15.283466 9854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Bootstrap starting.
I20260929 01:01:15.287043 9854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:15.288245 9854 log.cc:824] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:15.291774 9854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: No bootstrap required, opened a new log
I20260929 01:01:15.300112 9854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } }
I20260929 01:01:15.300539 9854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:15.300676 9854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e52fc8b85e45c4a7348d71e6f3911a, State: Initialized, Role: FOLLOWER
I20260929 01:01:15.301388 9854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } }
I20260929 01:01:15.302749 9861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } } }
I20260929 01:01:15.303143 9861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:15.303717 9854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:01:15.315392 9872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:15.315675 9872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:15.317170 9787 master_runner.cc:428] Detected that this master 65e52fc8b85e45c4a7348d71e6f3911a is joining an existing cluster
I20260929 01:01:15.317276 9787 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.21.253:33935
WARNING: no policy specified for oryx/127.3.21.252@KRBTEST.COM; defaulting to no policy
I20260929 01:01:15.342361 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.322846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47310)
0929 01:01:15.323652 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.323676 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.323825 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.324290 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.324295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.325003 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:15.325408 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.325421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.327438 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.327444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.329118 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.329133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.330518 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.330525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.330710 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.331528 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:15.331552 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:15.332915 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:15.338865 (+ 5950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.338873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.338882 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.339364 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.339926 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.339932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.339941 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.340094 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.341079 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:15.341085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:15.341247 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.341407 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.341776 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":923,"threads_started":1}
I20260929 01:01:15.342612 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.323505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.3.21.253:33935)
0929 01:01:15.323738 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.323746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.323773 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:01:15.323973 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.323978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.324045 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.324114 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.325608 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.327158 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.329281 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.330068 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.330164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.333170 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.333177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.333183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.333224 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.338682 (+ 5458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.339521 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.339529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.339533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.339768 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.340266 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.340275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.340281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.340931 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.341254 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.341598 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.342100 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":209,"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.252@KRBTEST.COM" created.
I20260929 01:01:15.345146 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.322832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.3.21.254:36493)
0929 01:01:15.323026 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.323033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.328051 (+ 5018us) server_negotiation.cc:407] Connection header received
0929 01:01:15.328314 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.328320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.328429 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.328500 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.329969 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.331124 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.333378 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.334429 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.334712 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.337788 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.337810 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.337818 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.337865 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.341916 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.342745 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.342751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.342754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.342962 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.343449 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.343453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.343456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.343812 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.343970 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.344502 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.344756 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:01:15.345218 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.323027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58526)
0929 01:01:15.327932 (+ 4905us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.327953 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.328163 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.328723 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.328730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.329439 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:15.329773 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.329786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.331393 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.331398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.333180 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.333195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.335054 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.335064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.335250 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.335974 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:15.335997 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:15.337516 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:15.342083 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.342088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.342092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.342595 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.343136 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.343140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.343144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.343302 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.344064 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:15.344069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:15.344241 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.344413 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.344756 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4757,"thread_start_us":4543,"threads_started":1}
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.254@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
I20260929 01:01:15.370409 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.353168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.3.21.253:33935)
0929 01:01:15.353409 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.353418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.353934 (+ 516us) server_negotiation.cc:407] Connection header received
0929 01:01:15.354269 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.354275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.354395 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.354632 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.355804 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.356669 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.358177 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.358835 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.359012 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.362962 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.362968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.362972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.363013 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.366779 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.367729 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.367736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.367744 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.367912 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.368448 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.368455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.368461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.368946 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.369151 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.369668 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.369956 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 01:01:15.370718 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.353086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47320)
0929 01:01:15.353781 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.353815 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.354049 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.354622 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.354630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.355388 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:15.355623 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.355634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.356840 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.356846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.357991 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.358004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.359033 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.359040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.359270 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.360079 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:15.360104 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:15.362701 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:15.366990 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.367010 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.367020 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.367558 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.368046 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.368053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.368057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.368278 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.369118 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:15.369128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:15.369334 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.369561 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.369957 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":325,"threads_started":1}
I20260929 01:01:15.373498 9323 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.21.253:33935: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
member_type: NON_VOTER
last_known_addr {
host: "127.3.21.253"
port: 33935
}
attrs {
promote: true
}
}
dest_uuid: "59123c9daf244e67983512edc364a35a"
cas_config_opid_index: -1
I20260929 01:01:15.374819 9323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } }
I20260929 01:01:15.379335 9885 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 65e52fc8b85e45c4a7348d71e6f3911a (127.3.21.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } }
I20260929 01:01:15.381145 9829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:15.385296 9885 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.21.253:33935
I20260929 01:01:15.386336 9787 master.cc:537] Master@127.3.21.253:33935 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.252.keytab.
Entry for principal oryx/127.3.21.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.252.keytab.
I20260929 01:01:15.388077 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:37453
--webserver_interface=127.3.21.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.252.keytab
--principal=oryx/127.3.21.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:15.392378 9884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } }
I20260929 01:01:15.392772 9884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:15.394627 9829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Refusing update from remote peer 59123c9daf244e67983512edc364a35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:01:15.396203 9885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:01:15.399892 9309 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 01:01:15.401901 9787 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:15.402561 9787 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:15.402801 9787 tablet_replica.cc:330] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: stopping tablet replica
W20260929 01:01:15.411113 9309 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a -> Peer 65e52fc8b85e45c4a7348d71e6f3911a (127.3.21.253:33935): Couldn't send request to peer 65e52fc8b85e45c4a7348d71e6f3911a. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:01:15.422662 9787 master.cc:552] Master@127.3.21.253:33935 shutdown complete.
I20260929 01:01:15.422770 9787 master_runner.cc:305] Clearing existing system tablet
I20260929 01:01:15.423255 9787 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:15.423369 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:15.426608 9787 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:15.428418 9890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:15.428791 9787 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:15.429066 9787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:15.429152 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "692fe48e139c55675689fe57256de3b6"
server_key_iv: "9aeefac840623a64ef0db5fc90efe092"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.434685 9787 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:15.442143 9787 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:01:15.442373 9787 log.cc:1196] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 01:01:15.442867 9787 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Deleting consensus metadata
I20260929 01:01:15.443473 9787 master_runner.cc:315] Copying system tablet from 127.3.21.254:36493
I20260929 01:01:15.446662 9787 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: tablet copy: Beginning tablet copy session from remote peer at address 127.3.21.254:36493
I20260929 01:01:15.468077 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.447676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.3.21.254:36493)
0929 01:01:15.447890 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.447897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.451296 (+ 3399us) server_negotiation.cc:407] Connection header received
0929 01:01:15.451505 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.451514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.451642 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.451805 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.452973 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.454198 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.456392 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.457120 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.457275 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.460004 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:15.460011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:15.460016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:15.460058 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:15.464408 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.465507 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.465515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.465522 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.465725 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:15.466260 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:15.466266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:15.466272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:15.466707 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:15.466862 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.467409 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.467659 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:01:15.468210 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.447605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58534)
0929 01:01:15.451109 (+ 3504us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.451140 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.451342 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.451840 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.451846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.452452 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:15.452769 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.452784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.454385 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.454392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.456181 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.456200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.457318 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.457325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.457536 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.458368 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:15.458423 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:15.459738 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:15.464631 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.464638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.464643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.465312 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.465892 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:15.465898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:15.465903 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:15.466106 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:15.466844 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:15.466850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:15.467074 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.467247 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.467674 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3295,"thread_start_us":3092,"threads_started":1}
I20260929 01:01:15.469703 9343 tablet_copy_service.cc:140] P 59123c9daf244e67983512edc364a35a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 65e52fc8b85e45c4a7348d71e6f3911a ({username='oryx', principal='oryx/127.3.21.253@KRBTEST.COM'} at 127.0.0.1:58534)
I20260929 01:01:15.470072 9343 tablet_copy_service.cc:161] P 59123c9daf244e67983512edc364a35a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 65e52fc8b85e45c4a7348d71e6f3911a at {username='oryx', principal='oryx/127.3.21.253@KRBTEST.COM'} at 127.0.0.1:58534: session id = 65e52fc8b85e45c4a7348d71e6f3911a-00000000000000000000000000000000
I20260929 01:01:15.474618 9343 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:01:15.477615 9787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:01:15.486518 9787 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: tablet copy: Starting download of 0 data blocks...
I20260929 01:01:15.486913 9787 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: tablet copy: Starting download of 1 WAL segments...
I20260929 01:01:15.490298 9787 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:01:15.495826 9343 tablet_copy_service.cc:342] P 59123c9daf244e67983512edc364a35a: Request end of tablet copy session 65e52fc8b85e45c4a7348d71e6f3911a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.21.253@KRBTEST.COM'} at 127.0.0.1:58534
I20260929 01:01:15.496091 9343 tablet_copy_service.cc:433] P 59123c9daf244e67983512edc364a35a: ending tablet copy session 65e52fc8b85e45c4a7348d71e6f3911a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 65e52fc8b85e45c4a7348d71e6f3911a
I20260929 01:01:15.511089 9787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:15.514793 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.519496 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.527329 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:15.527705 9787 server_base.cc:1034] running on GCE node
I20260929 01:01:15.528103 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:15.528924 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:15.530239 9787 hybrid_clock.cc:648] HybridClock initialized: now 1790643675530169 us; error 36 us; skew 500 ppm
I20260929 01:01:15.530539 9787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:15.532917 9787 init.cc:377] Logged in from keytab as oryx/127.3.21.253@KRBTEST.COM (short username oryx)
I20260929 01:01:15.535750 9787 webserver.cc:460] Webserver started at http://127.3.21.253:33619/ using document root <none> and password file <none>
I20260929 01:01:15.536265 9787 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:15.536365 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:15.540047 9787 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:01:15.542222 9907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:15.543058 9787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:01:15.543269 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "692fe48e139c55675689fe57256de3b6"
server_key_iv: "9aeefac840623a64ef0db5fc90efe092"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.543529 9787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:15.561379 9787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:15.566634 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:15.601656 9787 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.253:33935
I20260929 01:01:15.601696 9958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.253:33935 every 8 connection(s)
I20260929 01:01:15.602949 9787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
I20260929 01:01:15.608821 9959 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:01:15.615944 9959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Bootstrap starting.
I20260929 01:01:15.639138 9959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:01:15.639777 9959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: Bootstrap complete.
I20260929 01:01:15.641197 9959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } }
I20260929 01:01:15.641587 9959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65e52fc8b85e45c4a7348d71e6f3911a, State: Initialized, Role: LEARNER
I20260929 01:01:15.641913 9959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } }
I20260929 01:01:15.642897 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } }
I20260929 01:01:15.643309 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: LEARNER
I20260929 01:01:15.643376 9959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:15.663995 9973 catalog_manager.cc:1259] Loaded cluster ID: f310d5b134134361914cb13b4b9ab2cf
I20260929 01:01:15.664143 9973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: loading cluster ID for follower catalog manager: success
I20260929 01:01:15.667940 9973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: acquiring CA information for follower catalog manager: success
I20260929 01:01:15.670809 9973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:01:15.684003 9781 heartbeater.cc:503] Master 127.3.21.254:36493 was elected leader, sending a full tablet report...
W20260929 01:01:15.848146 9887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:15.848503 9887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:15.848675 9887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:15.858441 9887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:15.858735 9887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:15.858844 9887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:15.858964 9887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:15.859032 9887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:15.869763 9887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.252.keytab
--principal=oryx/127.3.21.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.21.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:15.871469 9887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:15.872916 9887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:15.887454 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.887519 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:15.887544 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:15.888274 9887 server_base.cc:1034] running on GCE node
I20260929 01:01:15.889117 9887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:15.890818 9887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:15.892055 9887 hybrid_clock.cc:648] HybridClock initialized: now 1790643675892016 us; error 41 us; skew 500 ppm
I20260929 01:01:15.892549 9887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:15 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:15.896787 9887 init.cc:377] Logged in from keytab as oryx/127.3.21.252@KRBTEST.COM (short username oryx)
I20260929 01:01:15.898942 9887 webserver.cc:460] Webserver started at http://127.3.21.252:35253/ using document root <none> and password file <none>
I20260929 01:01:15.899761 9887 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:15.899930 9887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:15.900257 9887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:15.903736 9887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/instance:
uuid: "16375c11274e45969111ed5f425adc24"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "9574f03c14302203dc48a86f4b07b192"
server_key_iv: "eb8742602f9a2b1e636c5eaa70cc40d7"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.904645 9887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal/instance:
uuid: "16375c11274e45969111ed5f425adc24"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "9574f03c14302203dc48a86f4b07b192"
server_key_iv: "eb8742602f9a2b1e636c5eaa70cc40d7"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.912096 9887 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:01:15.917357 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:15.919495 9887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:01:15.919766 9887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
uuid: "16375c11274e45969111ed5f425adc24"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "9574f03c14302203dc48a86f4b07b192"
server_key_iv: "eb8742602f9a2b1e636c5eaa70cc40d7"
server_key_version: "encryptionkey@0"
I20260929 01:01:15.920053 9887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:15.942137 9887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:15.948413 9887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:15.948737 9887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:15.970883 9887 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.252:37453
I20260929 01:01:15.970921 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.252:37453 every 8 connection(s)
I20260929 01:01:15.972544 9887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
I20260929 01:01:15.979877 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 9887
I20260929 01:01:15.986903 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:15.997462 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.974748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.3.21.253:33935)
0929 01:01:15.988491 (+ 13743us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.988499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.988525 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:01:15.988615 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.988622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.988729 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.988864 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:15.989395 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.991122 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.994115 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.996187 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.996360 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.996681 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:15.996963 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 01:01:15.997110 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13568,"thread_start_us":13440,"threads_started":1}
I20260929 01:01:15.997478 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.974563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47324)
0929 01:01:15.986437 (+ 11874us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:15.986457 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:15.987430 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:15.988953 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:15.988958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:15.988992 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:15.989209 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.989223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.991648 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.991666 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.993758 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:15.993783 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.996544 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.996552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:15.996749 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:15.996803 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 01:01:15.996932 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:01:15.997094 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11730,"thread_start_us":12088,"threads_started":1}
I20260929 01:01:15.999094 9934 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 01:01:15.999976 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59123c9daf244e67983512edc364a35a. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } }
I20260929 01:01:16.000232 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: LEARNER
I20260929 01:01:15.999217 10041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.252@KRBTEST.COM
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.252@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:16.019091 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:15.987352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.3.21.252:37453)
0929 01:01:15.988948 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:15.988966 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:15.989021 (+ 55us) server_negotiation.cc:407] Connection header received
0929 01:01:15.991408 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:15.991436 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:15.991936 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:15.992398 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:15.995172 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:15.997831 (+ 2659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:15.999912 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.001718 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.002098 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.008979 (+ 6881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.008989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.009002 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.009059 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.013541 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.014740 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.014753 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.014760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.015003 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.015652 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.015657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.015659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.016462 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.016817 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.017386 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.017853 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":1063,"threads_started":1}
I20260929 01:01:16.022336 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.002551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58550)
0929 01:01:16.003439 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.003499 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.003696 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.004577 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.004585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.005690 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.006169 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.006185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.007738 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.007744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.009257 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.009271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.010931 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.010944 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.011152 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.012015 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.012035 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.014925 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.018883 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.018895 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.018903 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.019569 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.020122 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.020128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.020131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.020295 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.021297 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.021303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.021473 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.021665 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.021949 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":447,"threads_started":1}
I20260929 01:01:16.022535 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.002685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.3.21.254:36493)
0929 01:01:16.004207 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.004213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.004239 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:01:16.004297 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.004301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.004365 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.004490 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.006401 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.007551 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.009444 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.010633 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.010801 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.015201 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.015207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.015210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.015248 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.018720 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.019729 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.019735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.019737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.019967 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.020633 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.020638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.020641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.021048 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.021210 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.021812 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.022044 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":1196,"threads_started":1}
I20260929 01:01:16.024056 10041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.252@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
I20260929 01:01:16.040407 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.024898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47330)
0929 01:01:16.025160 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.025184 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.025402 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.025882 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.025886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.026355 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.026613 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.026625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.027784 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.027788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.028811 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.028823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.029646 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.029651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.029845 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.030596 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.030613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.033403 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.037590 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.037595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.037597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.038074 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.038689 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.038696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.038699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.038847 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.039410 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.039415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.039564 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.039721 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.040055 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 01:01:16.040413 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.025169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.3.21.253:33935)
0929 01:01:16.025374 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.025380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.025402 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:01:16.025512 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.025516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.025576 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.025662 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.026774 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.027636 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.028978 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.029459 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.029570 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.033632 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.033638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.033641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.033678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.037389 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.038256 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.038263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.038267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.038568 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.038966 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.038972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.038976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.039277 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.039451 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.039808 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.040055 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:01:16.041716 10041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643675, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.252@KRBTEST.COM for oryx/127.3.21.252@KRBTEST.COM
I20260929 01:01:16.057216 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.042396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.0.0.1:50636)
0929 01:01:16.042657 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.042673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.042755 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.043072 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.043075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.043579 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.043722 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.043730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.045328 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.045389 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.046264 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.046275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.047321 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.047328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.047576 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.048467 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.048486 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.051106 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.054458 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.054462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.054465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.055037 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.055521 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.055525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.055528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.055680 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.056379 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.056383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.056571 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.056687 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.056891 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 01:01:16.057265 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.042550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.3.21.252:37453)
0929 01:01:16.042719 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.042723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.042739 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:16.042860 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.042863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.042917 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.042997 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.043889 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.045130 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.046442 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.047031 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.047133 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.051426 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.051433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.051436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.051471 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.054283 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.055154 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.055158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.055160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.055289 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.055837 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.055843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.055846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.056135 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.056254 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.056821 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.057029 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:01:16.063441 10041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Bootstrap starting.
I20260929 01:01:16.066915 10041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:16.068027 10041 log.cc:824] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:16.071673 10041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: No bootstrap required, opened a new log
I20260929 01:01:16.080160 10041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } }
I20260929 01:01:16.080565 10041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:16.080714 10041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16375c11274e45969111ed5f425adc24, State: Initialized, Role: FOLLOWER
I20260929 01:01:16.081421 10041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } }
I20260929 01:01:16.082423 10050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } } }
I20260929 01:01:16.082841 10050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:16.083706 10041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:01:16.097138 10061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:16.097358 10061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:16.102437 9887 master_runner.cc:428] Detected that this master 16375c11274e45969111ed5f425adc24 is joining an existing cluster
I20260929 01:01:16.102608 9887 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.21.252:37453
I20260929 01:01:16.129148 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9385
I20260929 01:01:16.138947 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.108353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.3.21.254:36493)
0929 01:01:16.108540 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.108546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.109534 (+ 988us) server_negotiation.cc:407] Connection header received
0929 01:01:16.109890 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.109898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.110009 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.110121 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.112045 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.113621 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.116676 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.117739 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.117888 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.124867 (+ 6979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.124877 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.124881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.124948 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.135587 (+ 10639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.136507 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.136513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.136517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.136695 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.137164 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.137169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.137173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.137536 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.137678 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.138346 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.138642 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:01:16.138942 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.108274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.3.21.253:33935)
0929 01:01:16.108533 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.108542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.110430 (+ 1888us) server_negotiation.cc:407] Connection header received
0929 01:01:16.110798 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.110803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.110887 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.110963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.112550 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.113662 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.116170 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.117331 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.117445 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.124899 (+ 7454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.124908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.124915 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.124979 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.134896 (+ 9917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.135740 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.135746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.135752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.135932 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.136451 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.136461 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.136467 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.136887 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.137040 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.138216 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.138534 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 01:01:16.139801 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.109071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47342)
0929 01:01:16.110328 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.110346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.110563 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.111179 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.111183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.111966 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.112359 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.112371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.113879 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.113885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.115928 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.115950 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.117816 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.117824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.118036 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.118896 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.118928 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.124568 (+ 5640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.135031 (+ 10463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.135037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.135043 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.135556 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.136093 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.136100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.136106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.136294 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.137124 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.137129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.137393 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.137561 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.137961 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":896,"threads_started":1}
I20260929 01:01:16.140856 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.108491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58562)
0929 01:01:16.109385 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.109407 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.109623 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.110346 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.110351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.111282 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.111802 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.111820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.113914 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.113923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.116366 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.116387 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.118291 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.118298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.118584 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.119303 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.119336 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.124503 (+ 5167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.135755 (+ 11252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.135760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.135764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.136351 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.136855 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.136860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.136863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.137028 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.137767 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.137774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.138028 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.139600 (+ 1572us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.140423 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":557,"threads_started":1}
I20260929 01:01:16.143483 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.110855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.0.0.1:50644)
0929 01:01:16.119352 (+ 8497us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.119376 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.119799 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.120557 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.120563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.124771 (+ 4208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.125271 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.125287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.128997 (+ 3710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.129005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.131633 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.131656 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.133570 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.133576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.133807 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.134509 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.134534 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.135718 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.140121 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.140125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.140129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.140693 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.141112 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.141117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.141120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.141262 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.141981 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.141986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.142146 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.142287 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.142656 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8296,"thread_start_us":8626,"threads_started":1}
I20260929 01:01:16.144450 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.110510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.3.21.252:37453)
0929 01:01:16.110712 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.110720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.119459 (+ 8739us) server_negotiation.cc:407] Connection header received
0929 01:01:16.120193 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.120201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.120284 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.120359 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.125476 (+ 5117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.126669 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.131925 (+ 5256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.132999 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.133280 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.135924 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.135930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.135935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.135973 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.139982 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.140852 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.140857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.140861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.140996 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.141379 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.141383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.141387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.141757 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.141932 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.143823 (+ 1891us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.144045 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.193@KRBTEST.COM
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.254@KRBTEST.COM for oryx/127.3.21.252@KRBTEST.COM
I20260929 01:01:16.157325 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.130468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.3.21.193:33573)
0929 01:01:16.131465 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.131478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.131513 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:16.131595 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.131606 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.131967 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.134025 (+ 2058us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.134322 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.136038 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.138159 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.139199 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.146882 (+ 7683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.148804 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.148824 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.148847 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.148891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.153123 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.154241 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.154251 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.154255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.154439 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.155333 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.155339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.155342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.155830 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.156136 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.156558 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.156869 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":538,"threads_started":1}
I20260929 01:01:16.159166 9508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:16.166551 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.147734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.0.0.1:50650)
0929 01:01:16.148037 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.148056 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.148165 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.148529 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.148536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.149007 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.149169 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.149178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.150632 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.150637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.151736 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.151749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.152610 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.152617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.152810 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.153474 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.153492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.156275 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.163535 (+ 7260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.163540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.163544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.163996 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.164507 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.164515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.164522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.164694 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.165443 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.165449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.165623 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.165786 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.166114 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260929 01:01:16.166599 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.147943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.3.21.252:37453)
0929 01:01:16.148129 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.148134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.148150 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:16.148306 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.148320 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.148394 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.148505 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.149323 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.150406 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.151905 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.152440 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.152576 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.158690 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.158698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.158703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.158751 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.163287 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.164153 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.164159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.164163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.164363 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.164845 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.164851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.164855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.165245 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.165465 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.165916 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.166160 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:01:16.168663 9323 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.21.252:37453: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "16375c11274e45969111ed5f425adc24"
member_type: NON_VOTER
last_known_addr {
host: "127.3.21.252"
port: 37453
}
attrs {
promote: true
}
}
dest_uuid: "59123c9daf244e67983512edc364a35a"
cas_config_opid_index: 5
I20260929 01:01:16.169911 9323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } }
I20260929 01:01:16.172369 10040 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 16375c11274e45969111ed5f425adc24 (127.3.21.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.176420 10040 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.21.252:37453
I20260929 01:01:16.178181 9934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 LEARNER]: Refusing update from remote peer 59123c9daf244e67983512edc364a35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:01:16.178609 10015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:16.183089 9887 master.cc:537] Master@127.3.21.252:37453 shutting down...
I20260929 01:01:16.185432 10015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Refusing update from remote peer 59123c9daf244e67983512edc364a35a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:01:16.197094 10040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.197623 10040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:16.198252 10040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:01:16.198819 10040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:16.207170 9934 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 16375c11274e45969111ed5f425adc24 (127.3.21.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.211159 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: NON_VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: true } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.211486 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: LEARNER
I20260929 01:01:16.213121 10040 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: attempting to promote NON_VOTER 65e52fc8b85e45c4a7348d71e6f3911a to VOTER
W20260929 01:01:16.213075 9994 tablet.cc:2369] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:16.213874 10040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } }
I20260929 01:01:16.217748 9887 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:16.218196 9934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 LEARNER]: Refusing update from remote peer 59123c9daf244e67983512edc364a35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 01:01:16.218403 9887 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:16.218705 9887 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: stopping tablet replica
I20260929 01:01:16.224664 10073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:01:16.227806 9887 master.cc:552] Master@127.3.21.252:37453 shutdown complete.
I20260929 01:01:16.227975 9887 master_runner.cc:305] Clearing existing system tablet
I20260929 01:01:16.228461 9887 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:16.228636 9887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:16.228354 10074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65e52fc8b85e45c4a7348d71e6f3911a (127.3.21.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.233491 10040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.233875 10040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:16.234702 10073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.235090 10073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:16.236590 9934 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 65e52fc8b85e45c4a7348d71e6f3911a (127.3.21.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.238703 9887 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.000s
I20260929 01:01:16.241748 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.242121 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:16.244814 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.244069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.0.0.1:50666)
0929 01:01:16.244319 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.244513 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:37453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:01:16.245301 9309 consensus_peers.cc:487] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a -> Peer 16375c11274e45969111ed5f425adc24 (127.3.21.252:37453): Couldn't send request to peer 16375c11274e45969111ed5f425adc24. Status: Network error: Client connection negotiation failed: client connection to 127.3.21.252:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:01:16.246941 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:16.247507 9887 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:16.247903 9887 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.002s
I20260929 01:01:16.248049 9887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
uuid: "16375c11274e45969111ed5f425adc24"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "9574f03c14302203dc48a86f4b07b192"
server_key_iv: "eb8742602f9a2b1e636c5eaa70cc40d7"
server_key_version: "encryptionkey@0"
I20260929 01:01:16.251952 9887 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:16.257882 9887 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:01:16.258100 9887 log.cc:1196] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 01:01:16.258584 9887 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Deleting consensus metadata
I20260929 01:01:16.258845 9887 master_runner.cc:315] Copying system tablet from 127.3.21.254:36493
I20260929 01:01:16.261574 9887 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: tablet copy: Beginning tablet copy session from remote peer at address 127.3.21.254:36493
I20260929 01:01:16.279300 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.262692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.3.21.254:36493)
0929 01:01:16.263003 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.263013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.267141 (+ 4128us) server_negotiation.cc:407] Connection header received
0929 01:01:16.267388 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.267394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.267469 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.267586 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.268497 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.269253 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.270583 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.271024 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.271172 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.273625 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.273632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.273642 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.273676 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.276474 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.277179 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.277186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.277190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.277332 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.277721 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.277727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.277730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.278009 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.278173 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.278695 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.278920 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260929 01:01:16.279994 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.262405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58572)
0929 01:01:16.266991 (+ 4586us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.267017 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.267226 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.267609 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.267618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.268125 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.268342 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.268350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.269402 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.269408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.270404 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.270415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.271189 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.271194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.271415 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.272127 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.272144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.273388 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.276638 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.276644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.276648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.277044 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.277449 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.277455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.277458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.277608 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.278138 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.278143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.278320 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.278472 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.279244 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4450,"thread_start_us":4314,"threads_started":1}
I20260929 01:01:16.280689 9343 tablet_copy_service.cc:140] P 59123c9daf244e67983512edc364a35a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 16375c11274e45969111ed5f425adc24 ({username='oryx', principal='oryx/127.3.21.252@KRBTEST.COM'} at 127.0.0.1:58572)
I20260929 01:01:16.280867 9343 tablet_copy_service.cc:161] P 59123c9daf244e67983512edc364a35a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 16375c11274e45969111ed5f425adc24 at {username='oryx', principal='oryx/127.3.21.252@KRBTEST.COM'} at 127.0.0.1:58572: session id = 16375c11274e45969111ed5f425adc24-00000000000000000000000000000000
I20260929 01:01:16.283316 9343 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:01:16.285327 9887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:01:16.291481 9887 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: tablet copy: Starting download of 0 data blocks...
I20260929 01:01:16.291833 9887 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: tablet copy: Starting download of 1 WAL segments...
I20260929 01:01:16.294279 9887 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:01:16.297879 9343 tablet_copy_service.cc:342] P 59123c9daf244e67983512edc364a35a: Request end of tablet copy session 16375c11274e45969111ed5f425adc24-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.21.252@KRBTEST.COM'} at 127.0.0.1:58572
I20260929 01:01:16.298056 9343 tablet_copy_service.cc:433] P 59123c9daf244e67983512edc364a35a: ending tablet copy session 16375c11274e45969111ed5f425adc24-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 16375c11274e45969111ed5f425adc24
I20260929 01:01:16.311234 9887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:16.315932 10086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:16.317297 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:16.321220 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9385
W20260929 01:01:16.325765 10089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:16.325883 9887 server_base.cc:1034] running on GCE node
I20260929 01:01:16.326314 9887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:16.330830 9887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:16.332110 9887 hybrid_clock.cc:648] HybridClock initialized: now 1790643676332035 us; error 79 us; skew 500 ppm
I20260929 01:01:16.332458 9887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:16.334903 9887 init.cc:377] Logged in from keytab as oryx/127.3.21.252@KRBTEST.COM (short username oryx)
I20260929 01:01:16.338544 9887 webserver.cc:460] Webserver started at http://127.3.21.252:33507/ using document root <none> and password file <none>
I20260929 01:01:16.338929 9887 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:16.339041 9887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:16.343978 9887 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:01:16.346517 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:33573
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:36493
--webserver_port=40525
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:16.346747 10096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:16.347289 9887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:01:16.347430 9887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
uuid: "16375c11274e45969111ed5f425adc24"
format_stamp: "Formatted at 2026-09-29 01:01:15 on dist-test-slave-k0b7"
server_key: "9574f03c14302203dc48a86f4b07b192"
server_key_iv: "eb8742602f9a2b1e636c5eaa70cc40d7"
server_key_version: "encryptionkey@0"
I20260929 01:01:16.347640 9887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:16.371511 9887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:16.375754 9887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:16.397408 9887 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.252:37453
I20260929 01:01:16.397466 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.252:37453 every 8 connection(s)
I20260929 01:01:16.398764 9887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
I20260929 01:01:16.404693 10149 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:01:16.410255 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Bootstrap starting.
I20260929 01:01:16.430157 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:01:16.430763 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Bootstrap complete.
I20260929 01:01:16.431831 10149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } }
I20260929 01:01:16.432127 10149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16375c11274e45969111ed5f425adc24, State: Initialized, Role: LEARNER
I20260929 01:01:16.432410 10149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } }
I20260929 01:01:16.433560 10149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:16.433529 10152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.434075 10152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:01:16.442878 10163 catalog_manager.cc:1259] Loaded cluster ID: f310d5b134134361914cb13b4b9ab2cf
I20260929 01:01:16.443009 10163 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: loading cluster ID for follower catalog manager: success
I20260929 01:01:16.446780 10163 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: acquiring CA information for follower catalog manager: success
I20260929 01:01:16.449693 10163 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:01:16.669369 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.661579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.3.21.252:37453)
0929 01:01:16.662186 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.662190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.662214 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:01:16.662442 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.662447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.662592 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.662704 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:16.663218 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.664489 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.666978 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.668067 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.668330 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.668521 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.668975 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.669092 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":348,"threads_started":1}
I20260929 01:01:16.669416 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.661314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.0.0.1:50678)
0929 01:01:16.662079 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.662106 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.662255 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.662752 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.662759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.662788 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:16.663023 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.663037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.664688 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.664694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.666756 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.666776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.668289 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.668297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.668686 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.668788 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.668923 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.669112 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 01:01:16.670951 10124 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 01:01:16.671622 10152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59123c9daf244e67983512edc364a35a. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:16.671902 10152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:01:16.739970 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:16.740329 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:16.740509 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:16.748638 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:16.748765 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:16.748939 10101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:16.760221 10101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:33573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=40525
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:16.761682 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:16.763646 10101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:16.781000 10170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:16.781057 10169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:16.781402 10172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:16.782546 10101 server_base.cc:1034] running on GCE node
I20260929 01:01:16.783254 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:16.784662 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:16.785957 10101 hybrid_clock.cc:648] HybridClock initialized: now 1790643676785906 us; error 84 us; skew 500 ppm
I20260929 01:01:16.786369 10101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:16.790316 10101 init.cc:377] Logged in from keytab as oryx/127.3.21.193@KRBTEST.COM (short username oryx)
I20260929 01:01:16.792119 10101 webserver.cc:460] Webserver started at http://127.3.21.193:40525/ using document root <none> and password file <none>
I20260929 01:01:16.792938 10101 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:16.793120 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:16.799782 10101 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:01:16.803247 10179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:16.805248 10101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:16.805511 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "b775d706489f4d0ea3084db13ee49cb9"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "f143b7bf3446a55afa96571c3d3fdfbf"
server_key_iv: "723c2a51f40838ba610b262fc18b6f29"
server_key_version: "encryptionkey@0"
I20260929 01:01:16.806809 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:16.823189 10101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:16.828138 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:16.828471 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:16.829668 10101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:16.831319 10101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:16.831461 10101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:16.831609 10101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:16.832140 10101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:16.874986 10101 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:33573
I20260929 01:01:16.875015 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:33573 every 8 connection(s)
I20260929 01:01:16.876600 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:16.883381 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 10101
I20260929 01:01:16.883852 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9519
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643676, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.252@KRBTEST.COM
I20260929 01:01:16.906905 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.881214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:39751 (local address 127.3.21.254:36493)
0929 01:01:16.884284 (+ 3070us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.884290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.884311 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:01:16.885447 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.885453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.885537 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.885657 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.887913 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.889093 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.891970 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.892369 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.892545 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.899374 (+ 6829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.899380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.899383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.899427 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.902967 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.904094 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.904099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.904103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.904231 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.904778 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.904783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.904786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.905089 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.905258 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.906250 (+ 992us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.906523 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2947,"thread_start_us":2835,"threads_started":1}
I20260929 01:01:16.908766 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.880467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.193:39751)
0929 01:01:16.881889 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.881907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.885093 (+ 3186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.885994 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.886021 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.886962 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.887703 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.887724 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.889246 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.889252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.891078 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.891092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.892692 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.892704 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.893072 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.894150 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.894190 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.899085 (+ 4895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.903125 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.903137 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.903162 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.903972 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.904357 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.904362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.904367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.904655 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.905233 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.905242 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.905802 (+ 560us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.906147 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.906873 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":87,"spinlock_wait_cycles":2688,"thread_start_us":710,"threads_started":1}
Sep 29 01:01:16 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.194@KRBTEST.COM
I20260929 01:01:16.910954 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.881106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:55439 (local address 127.3.21.253:33935)
0929 01:01:16.882821 (+ 1715us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.882828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.882858 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:01:16.888032 (+ 5174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.888038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.888153 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.888238 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.890979 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.891824 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.895866 (+ 4042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.896404 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.896497 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.901897 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.901903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.901907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.901943 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.906459 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.907575 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.907624 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.907632 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.907798 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.908340 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.908347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.908352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.908754 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.909027 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.910075 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.910336 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583,"thread_start_us":2142,"threads_started":1}
I20260929 01:01:16.912024 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.880467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.3.21.193:55439)
0929 01:01:16.881757 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.881829 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.885560 (+ 3731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.888512 (+ 2952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.888517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.888998 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.890739 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.890756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.892783 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.892788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.895622 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.895648 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.896761 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.896766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.896889 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.898032 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.898063 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.901635 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.906695 (+ 5060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.906702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.906707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.907384 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.907983 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.907988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.907993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.908199 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.909548 (+ 1349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.909556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.909761 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.911155 (+ 1394us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.911512 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":522,"spinlock_wait_cycles":97408,"thread_start_us":645,"threads_started":1}
I20260929 01:01:16.912884 10293 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36493
I20260929 01:01:16.913302 10293 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:16.913512 10294 heartbeater.cc:347] Connected to a master server at 127.3.21.253:33935
I20260929 01:01:16.913657 10294 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:16.914736 10294 heartbeater.cc:511] Master 127.3.21.253:33935 requested a full tablet report, sending...
I20260929 01:01:16.914796 10293 heartbeater.cc:511] Master 127.3.21.254:36493 requested a full tablet report, sending...
I20260929 01:01:16.916101 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.884939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.3.21.193:49139)
0929 01:01:16.894121 (+ 9182us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:16.894143 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:16.894284 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:16.894820 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:16.894824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:16.895548 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:16.895999 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.896014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.897854 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.897860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.899713 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:16.899729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.901077 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.901086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:16.901300 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.903263 (+ 1963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:16.903288 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:16.907980 (+ 4692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:16.912657 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.912664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.912668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.913327 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.914011 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:16.914015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:16.914019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:16.914225 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:16.914962 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:16.914968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:16.915128 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 01:01:16.915333 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 01:01:16.915661 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8662,"thread_start_us":8475,"threads_started":1}
I20260929 01:01:16.916944 9323 ts_manager.cc:194] Re-registered known tserver with Master: b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:16.917902 9924 ts_manager.cc:194] Registered new tserver with Master: b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:16.918071 9323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:39751
I20260929 01:01:16.919004 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.883063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:49139 (local address 127.3.21.252:37453)
0929 01:01:16.883248 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.883253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.894265 (+ 11012us) server_negotiation.cc:407] Connection header received
0929 01:01:16.894397 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.894402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.894509 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.894614 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.896202 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.897683 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.899902 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.900639 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.900747 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.908245 (+ 7498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.908252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.908257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.908299 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.912495 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.913524 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.913533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.913538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.913845 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.914334 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.914340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.914343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.914746 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.914902 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.915446 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.915672 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:01:16.920244 10292 heartbeater.cc:347] Connected to a master server at 127.3.21.252:37453
I20260929 01:01:16.920414 10292 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:16.921216 10292 heartbeater.cc:511] Master 127.3.21.252:37453 requested a full tablet report, sending...
I20260929 01:01:16.922652 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:16.884833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.3.21.194:44107)
0929 01:01:16.894266 (+ 9433us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:16.894275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:16.894308 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:01:16.894375 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:16.894391 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:16.894702 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:16.897602 (+ 2900us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:16.897888 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.899687 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.902110 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:16.903296 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:16.904143 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:16.911987 (+ 7844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:16.912013 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:16.912052 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:16.912110 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:16.916292 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.917247 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.917257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.917261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.917428 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:16.917847 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:16.917853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:16.917856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:16.918223 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:16.918602 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:16.921399 (+ 2797us) server_negotiation.cc:299] Negotiation successful
0929 01:01:16.922234 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9210,"thread_start_us":9057,"threads_started":1}
I20260929 01:01:16.923566 10114 ts_manager.cc:194] Registered new tserver with Master: b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:16.924206 9642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:17.015826 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9519
I20260929 01:01:17.033408 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:44107
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:36493
--webserver_port=34829
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:17.240729 10074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: NON_VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: true } } }
I20260929 01:01:17.241295 10074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:17.243431 10074 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: attempting to promote NON_VOTER 16375c11274e45969111ed5f425adc24 to VOTER
I20260929 01:01:17.244377 10074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } }
I20260929 01:01:17.246840 9934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Refusing update from remote peer 59123c9daf244e67983512edc364a35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 01:01:17.247227 10124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 LEARNER]: Refusing update from remote peer 59123c9daf244e67983512edc364a35a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 01:01:17.247659 10074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 01:01:17.248623 10074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 01:01:17.251609 10074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 16375c11274e45969111ed5f425adc24 (127.3.21.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.252140 9934 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 16375c11274e45969111ed5f425adc24 (127.3.21.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.256210 10074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.256584 10074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:17.256896 10074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.257160 10074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:17.256928 10311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.257505 10311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:17.259217 10306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.259558 10306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:17.259994 10124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 16375c11274e45969111ed5f425adc24 (127.3.21.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.263685 10312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59123c9daf244e67983512edc364a35a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:17.264880 10312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:01:17.423280 10304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:17.423619 10304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:17.423719 10304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:17.433233 10304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:17.433414 10304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:17.433558 10304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:17.445241 10304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:44107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=34829
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:17.446870 10304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:17.448365 10304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:17.464102 10317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:17.464211 10320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:17.464277 10318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:17.470201 10304 server_base.cc:1034] running on GCE node
I20260929 01:01:17.471109 10304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:17.472270 10304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:17.473533 10304 hybrid_clock.cc:648] HybridClock initialized: now 1790643677473487 us; error 48 us; skew 500 ppm
I20260929 01:01:17.473909 10304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:17 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643677, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:17.477718 10304 init.cc:377] Logged in from keytab as oryx/127.3.21.194@KRBTEST.COM (short username oryx)
I20260929 01:01:17.479463 10304 webserver.cc:460] Webserver started at http://127.3.21.194:34829/ using document root <none> and password file <none>
I20260929 01:01:17.480278 10304 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:17.480449 10304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:17.487697 10304 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:01:17.491609 10327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:17.493474 10304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:01:17.493690 10304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249"
format_stamp: "Formatted at 2026-09-29 01:01:13 on dist-test-slave-k0b7"
server_key: "93edca9da1bcf6c6d0c38f711bd01228"
server_key_iv: "33aa777197b0e8802db4dc870c82a1d8"
server_key_version: "encryptionkey@0"
I20260929 01:01:17.495386 10304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:17.518863 10304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:17.524030 10304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:17.524731 10304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:17.525982 10304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:17.527664 10304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:17.527797 10304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:17.527941 10304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:17.528033 10304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:17.567420 10304 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:44107
I20260929 01:01:17.567536 10439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:44107 every 8 connection(s)
I20260929 01:01:17.568992 10304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:17.569744 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 10304
I20260929 01:01:17.570148 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9653
Sep 29 01:01:17 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643677, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
Sep 29 01:01:17 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643677, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.252@KRBTEST.COM
Sep 29 01:01:17 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643677, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
Sep 29 01:01:17 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.195@KRBTEST.COM
I20260929 01:01:17.620030 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.579470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:52561 (local address 127.3.21.254:36493)
0929 01:01:17.587061 (+ 7591us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:17.587081 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:17.591596 (+ 4515us) server_negotiation.cc:407] Connection header received
0929 01:01:17.592938 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:17.592947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:17.593145 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:17.593269 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:17.597332 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.598525 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.601617 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.602291 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.602411 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.609310 (+ 6899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:17.609589 (+ 279us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:17.609597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:17.609670 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:17.614435 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.615637 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.615644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.615649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.615846 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.616437 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.616444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.616448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.617433 (+ 985us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:17.617605 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:17.619279 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0929 01:01:17.619657 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7396,"thread_start_us":7238,"threads_started":1}
I20260929 01:01:17.621778 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.578908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:52561)
0929 01:01:17.591459 (+ 12551us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:17.591483 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:17.592729 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:17.594000 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:17.594021 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:17.595437 (+ 1416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:17.597033 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:17.597070 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.598741 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.598747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:17.601408 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:17.601424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.602784 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.602790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:17.603093 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.604160 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:17.604199 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:17.608332 (+ 4133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:17.614693 (+ 6361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:17.614705 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:17.614728 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:17.615472 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:17.616015 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:17.616020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:17.616024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:17.616297 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:17.617740 (+ 1443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:17.617750 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:17.618449 (+ 699us) client_negotiation.cc:769] Sending connection context
0929 01:01:17.619443 (+ 994us) client_negotiation.cc:240] Negotiation successful
0929 01:01:17.620159 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12396,"mutex_wait_us":1,"thread_start_us":8431,"threads_started":1}
I20260929 01:01:17.623003 10447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.574761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:39121 (local address 127.3.21.253:33935)
0929 01:01:17.593396 (+ 18635us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:17.593405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:17.593431 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:01:17.593512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:17.593520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:17.593619 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:17.593764 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:17.597682 (+ 3918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.598807 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.604534 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.605225 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.605350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.613825 (+ 8475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:17.613830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:17.613833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:17.613888 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:17.618648 (+ 4760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.620138 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.620145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.620150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.620332 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.620932 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.620937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.620941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.621292 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:17.621469 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:17.622352 (+ 883us) server_negotiation.cc:299] Negotiation successful
0929 01:01:17.622653 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18462,"thread_start_us":19133,"threads_started":1}
I20260929 01:01:17.623760 10441 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36493
I20260929 01:01:17.623849 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.591815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:59945 (local address 127.3.21.252:37453)
0929 01:01:17.595670 (+ 3855us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:17.595676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:17.596230 (+ 554us) server_negotiation.cc:407] Connection header received
0929 01:01:17.596300 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:17.596305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:17.596386 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:17.598199 (+ 1813us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:17.598369 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.599532 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.602682 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.603405 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.603527 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.612832 (+ 9305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:17.612839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:17.612846 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:17.612911 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:17.618082 (+ 5171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.620674 (+ 2592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.620681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.620685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.620846 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.621428 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.621436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.621440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.621820 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:17.621956 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:17.622584 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 01:01:17.623461 (+ 877us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3707,"thread_start_us":3552,"threads_started":1}
I20260929 01:01:17.623874 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.574009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.3.21.194:39121)
0929 01:01:17.588006 (+ 13997us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:17.588084 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:17.591940 (+ 3856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:17.595979 (+ 4039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:17.595993 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:17.596546 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:17.597026 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:17.597063 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.599067 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.599073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:17.602442 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:17.602453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.606418 (+ 3965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.606424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:17.606669 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.608664 (+ 1995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:17.608694 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:17.612039 (+ 3345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:17.618828 (+ 6789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:17.618834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:17.618837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:17.619509 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:17.620593 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:17.620598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:17.620601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:17.620783 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:17.621531 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:17.621535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:17.621712 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 01:01:17.621867 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 01:01:17.623413 (+ 1546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13505,"mutex_wait_us":1541,"thread_start_us":16893,"threads_started":1}
I20260929 01:01:17.624148 10441 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:17.624899 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.594257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.3.21.194:59945)
0929 01:01:17.595923 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:17.595942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:17.596074 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:17.596550 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:17.596554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:17.597248 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:17.598034 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:17.598047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.599724 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.599731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:17.602179 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:17.602193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.603791 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.603801 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:17.604030 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.605678 (+ 1648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:17.605705 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:17.612554 (+ 6849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:17.619263 (+ 6709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:17.619268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:17.619272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:17.620517 (+ 1245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:17.620992 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:17.620997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:17.620999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:17.621285 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:17.622033 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:17.622037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:17.622194 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 01:01:17.622339 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 01:01:17.624173 (+ 1834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"mutex_wait_us":1482,"thread_start_us":1405,"threads_started":1}
I20260929 01:01:17.625316 10441 heartbeater.cc:511] Master 127.3.21.254:36493 requested a full tablet report, sending...
I20260929 01:01:17.625550 10442 heartbeater.cc:347] Connected to a master server at 127.3.21.253:33935
I20260929 01:01:17.625715 10442 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:17.625870 10440 heartbeater.cc:347] Connected to a master server at 127.3.21.252:37453
I20260929 01:01:17.626005 10440 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:17.626331 10442 heartbeater.cc:511] Master 127.3.21.253:33935 requested a full tablet report, sending...
I20260929 01:01:17.626547 10440 heartbeater.cc:511] Master 127.3.21.252:37453 requested a full tablet report, sending...
I20260929 01:01:17.627192 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:17.572056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.3.21.195:37327)
0929 01:01:17.598104 (+ 26048us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:17.598114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:17.598147 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:01:17.598223 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:17.598231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:17.598557 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:17.599481 (+ 924us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:17.601367 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.602945 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.604934 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:17.605991 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:17.607011 (+ 1020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:17.612935 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:17.612956 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:17.612980 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:17.613026 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:17.620846 (+ 7820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.622047 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.622057 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.622060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.622230 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:17.623212 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:17.623216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:17.623219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:17.623639 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:17.625923 (+ 2284us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:17.626218 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 01:01:17.626785 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25455,"thread_start_us":25255,"threads_started":1}
I20260929 01:01:17.627338 9323 ts_manager.cc:194] Re-registered known tserver with Master: 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107)
I20260929 01:01:17.627689 9924 ts_manager.cc:194] Registered new tserver with Master: 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107)
I20260929 01:01:17.627686 10114 ts_manager.cc:194] Registered new tserver with Master: 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107)
I20260929 01:01:17.628472 9323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:52561
I20260929 01:01:17.628883 9776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:17.723958 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9653
I20260929 01:01:17.743290 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:37327
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:36493
--webserver_port=41983
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:17.921509 10293 heartbeater.cc:503] Master 127.3.21.254:36493 was elected leader, sending a full tablet report...
W20260929 01:01:18.156615 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:18.156996 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:18.157115 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:18.165884 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:18.166038 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:18.166167 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:18.178347 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:37327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=41983
--tserver_master_addrs=127.3.21.254:36493,127.3.21.253:33935,127.3.21.252:37453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:18.180480 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:18.183019 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:18.205471 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:18.206221 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:18.205470 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:18.207039 10453 server_base.cc:1034] running on GCE node
I20260929 01:01:18.208006 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:18.209651 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:18.211074 10453 hybrid_clock.cc:648] HybridClock initialized: now 1790643678211004 us; error 57 us; skew 500 ppm
I20260929 01:01:18.211606 10453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:18 dist-test-slave-k0b7 krb5kdc[9278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643678, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:18.216549 10453 init.cc:377] Logged in from keytab as oryx/127.3.21.195@KRBTEST.COM (short username oryx)
I20260929 01:01:18.219074 10453 webserver.cc:460] Webserver started at http://127.3.21.195:41983/ using document root <none> and password file <none>
I20260929 01:01:18.220263 10453 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:18.220468 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:18.229738 10453 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:01:18.234216 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:18.236626 10453 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:01:18.236871 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "315db3dbd8e94f5db25b298148d6ce54"
format_stamp: "Formatted at 2026-09-29 01:01:14 on dist-test-slave-k0b7"
server_key: "c4e16659c36d5b0207ec01d5efc96f32"
server_key_iv: "766ff9402925236671f7997e39b0631b"
server_key_version: "encryptionkey@0"
I20260929 01:01:18.238533 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:18.254628 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:18.260562 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:18.261031 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:18.262326 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:18.264258 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:18.264407 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:18.264549 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:18.264640 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:18.313164 10453 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37327
I20260929 01:01:18.313257 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37327 every 8 connection(s)
I20260929 01:01:18.315083 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:18.321313 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 10453
Sep 29 01:01:18 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643678, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
Sep 29 01:01:18 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643678, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.253@KRBTEST.COM
Sep 29 01:01:18 dist-test-slave-k0b7 krb5kdc[9278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643678, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.252@KRBTEST.COM
I20260929 01:01:18.362989 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.318649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:44175 (local address 127.3.21.253:33935)
0929 01:01:18.319136 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.319141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.325572 (+ 6431us) server_negotiation.cc:407] Connection header received
0929 01:01:18.327129 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.327135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.327247 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.327375 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.331608 (+ 4233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.332810 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.336881 (+ 4071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.337595 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.337736 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.349560 (+ 11824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.349571 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.349579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.349630 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.354551 (+ 4921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.355800 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.355810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.355816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.356090 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.358008 (+ 1918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.358018 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.358024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.358559 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.358794 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.362027 (+ 3233us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.362385 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":242,"threads_started":1}
I20260929 01:01:18.364395 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.319834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:53227 (local address 127.3.21.254:36493)
0929 01:01:18.327572 (+ 7738us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.327578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.328934 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 01:01:18.329093 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.329098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.329180 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.329308 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.332062 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.334208 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.337323 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.338147 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.338336 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.347423 (+ 9087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.347433 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.347441 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.347514 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.352249 (+ 4735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.354024 (+ 1775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.354033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.354040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.354256 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.355836 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.355841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.355846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.356414 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.356621 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.363201 (+ 6580us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.363587 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7590,"thread_start_us":7652,"threads_started":1}
I20260929 01:01:18.364980 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.319829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:53227)
0929 01:01:18.328764 (+ 8935us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.328788 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.328957 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.329333 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.329337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.331560 (+ 2223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:18.331881 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.331893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.334746 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.334752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.337076 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.337093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.339044 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.339055 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.339558 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.340732 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:18.340770 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:18.346970 (+ 6200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:18.352519 (+ 5549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:18.352535 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:18.352567 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:18.353767 (+ 1200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:18.354441 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:18.354449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:18.354456 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:18.354815 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:18.357154 (+ 2339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:18.357166 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:18.357875 (+ 709us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.358317 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.359176 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8788,"mutex_wait_us":181,"thread_start_us":4706,"threads_started":1}
I20260929 01:01:18.365772 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.319263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.3.21.195:44175)
0929 01:01:18.321255 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.321344 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.326721 (+ 5377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.328489 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.328506 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.330208 (+ 1702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:18.331382 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.331412 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.333072 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.333079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.335959 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.335980 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.339026 (+ 3046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.339042 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.339560 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.343224 (+ 3664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:18.343273 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:18.349251 (+ 5978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:18.354972 (+ 5721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:18.354980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:18.354986 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:18.355638 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:18.356709 (+ 1071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:18.356716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:18.356723 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:18.356976 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:18.361248 (+ 4272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:18.361257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:18.361550 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.361770 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.362254 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"mutex_wait_us":157,"spinlock_wait_cycles":25472,"thread_start_us":9065,"threads_started":1}
I20260929 01:01:18.367177 10582 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36493
I20260929 01:01:18.367324 10583 heartbeater.cc:347] Connected to a master server at 127.3.21.253:33935
I20260929 01:01:18.367552 10583 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:18.367563 10582 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:18.368849 10583 heartbeater.cc:511] Master 127.3.21.253:33935 requested a full tablet report, sending...
I20260929 01:01:18.368860 10582 heartbeater.cc:511] Master 127.3.21.254:36493 requested a full tablet report, sending...
I20260929 01:01:18.372221 9323 ts_manager.cc:194] Re-registered known tserver with Master: 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195:37327)
I20260929 01:01:18.374210 9323 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:53227
I20260929 01:01:18.376863 9924 ts_manager.cc:194] Registered new tserver with Master: 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195:37327)
I20260929 01:01:18.381781 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.329237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.3.21.195:46681)
0929 01:01:18.331190 (+ 1953us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.331212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.331355 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.332067 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.332071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.332671 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:18.333019 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.333031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.337589 (+ 4558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.337594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.340284 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.340307 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.342584 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.342594 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.342870 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.345278 (+ 2408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:18.345309 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:18.350980 (+ 5671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:18.369175 (+ 18195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:18.369179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:18.369183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:18.370702 (+ 1519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:18.379585 (+ 8883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:18.379595 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:18.379601 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:18.379859 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:18.380561 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:18.380567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:18.380745 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.380924 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.381329 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1805,"thread_start_us":1637,"threads_started":1}
I20260929 01:01:18.381850 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.329225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:46681 (local address 127.3.21.252:37453)
0929 01:01:18.330708 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.330715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.331343 (+ 628us) server_negotiation.cc:407] Connection header received
0929 01:01:18.331513 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.331523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.331637 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.331775 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.333560 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.337412 (+ 3852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.340577 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.341323 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.354023 (+ 12700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.354289 (+ 266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.354297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.354302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.354336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.369024 (+ 14688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.379208 (+ 10184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.379218 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.379224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.379445 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.379985 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.379991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.379994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.380433 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.380604 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.381158 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.381447 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":1249,"threads_started":1}
I20260929 01:01:18.383014 10581 heartbeater.cc:347] Connected to a master server at 127.3.21.252:37453
I20260929 01:01:18.383186 10581 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:18.384083 10581 heartbeater.cc:511] Master 127.3.21.252:37453 requested a full tablet report, sending...
I20260929 01:01:18.386098 10114 ts_manager.cc:194] Registered new tserver with Master: 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195:37327)
I20260929 01:01:18.389074 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.339660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.3.21.254:36493)
0929 01:01:18.341160 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.341167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.341185 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:18.341246 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.341251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.341338 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.344584 (+ 3246us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.345094 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.346445 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.349244 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.356391 (+ 7147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.356548 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.369973 (+ 13425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.369982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.369989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.370040 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.378966 (+ 8926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.380703 (+ 1737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.380710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.380713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.380898 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.382376 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.382385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.382389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.382992 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.387500 (+ 4508us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.387958 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.388633 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":1251,"threads_started":1}
I20260929 01:01:18.393249 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.341736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.3.21.253:33935)
0929 01:01:18.363296 (+ 21560us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.363302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.366614 (+ 3312us) server_negotiation.cc:407] Connection header received
0929 01:01:18.367102 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.367107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.367215 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.367309 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.372181 (+ 4872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.373954 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.376156 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.377022 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.377178 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.383215 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.383222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.383228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.383272 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.388162 (+ 4890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.390268 (+ 2106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.390272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.390275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.390516 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.391062 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.391067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.391071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.391429 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.391717 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.392531 (+ 814us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.392824 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21453,"thread_start_us":23092,"threads_started":1}
I20260929 01:01:18.395313 9323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:01:18.407761 9323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:18.408574 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.366999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.3.21.252:37453)
0929 01:01:18.383317 (+ 16318us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.383323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.383939 (+ 616us) server_negotiation.cc:407] Connection header received
0929 01:01:18.384004 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.384010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.384116 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.384217 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.387199 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.388907 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.392229 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.392954 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.393094 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.397431 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.397438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.397443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.397491 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.404559 (+ 7068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.405849 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.405856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.405859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.406106 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.406619 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.406625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.406628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.407047 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.407409 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.407840 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.408127 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16203,"thread_start_us":19887,"threads_started":1}
I20260929 01:01:18.454716 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.435364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.3.21.193:33573)
0929 01:01:18.441409 (+ 6045us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.441422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.441457 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:18.441583 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.441594 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.445413 (+ 3819us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.445628 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:18.446738 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.448474 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.452084 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.453640 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.453818 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.454093 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.454252 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.454390 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5834,"thread_start_us":6271,"threads_started":1}
I20260929 01:01:18.454845 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.434801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:33573 (local address 127.0.0.1:43996)
0929 01:01:18.440743 (+ 5942us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.440765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.440902 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.445883 (+ 4981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.445888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.445913 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:18.446477 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.446515 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.448691 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.448699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.451920 (+ 3221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.451933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.453898 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.453905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.454083 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.454117 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.454200 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.454364 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5790,"mutex_wait_us":1,"thread_start_us":5857,"threads_started":1}
I20260929 01:01:18.459141 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.433790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:44107 (local address 127.0.0.1:45400)
0929 01:01:18.439483 (+ 5693us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.439506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.439671 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.445039 (+ 5368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.445044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.445079 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:18.446283 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.446301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.452667 (+ 6366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.452675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.454378 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.454390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.458249 (+ 3859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.458257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.458425 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.458460 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.458596 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.458751 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5532,"thread_start_us":17218,"threads_started":1}
I20260929 01:01:18.460022 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.434159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.3.21.194:44107)
0929 01:01:18.444345 (+ 10186us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.444355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.444389 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:01:18.444502 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.444510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.444773 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.448249 (+ 3476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:18.448508 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.450266 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.456179 (+ 5913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.457722 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.457860 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.458099 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.459525 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.459759 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9998,"thread_start_us":9897,"threads_started":1}
I20260929 01:01:18.460052 10227 tablet_service.cc:1467] Processing CreateTablet for tablet 4e945681d782472692cd317015680b30 (DEFAULT_TABLE table=test-table [id=3d81540f09ef423688765fc2e3194ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:18.461124 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.434985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37327 (local address 127.0.0.1:43046)
0929 01:01:18.451341 (+ 16356us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.451363 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.451494 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.452300 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.452304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.452332 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:18.452736 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.452747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.456035 (+ 3288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.456042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.458047 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.458064 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.460277 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.460282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.460416 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.460440 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.460568 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.460724 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16177,"thread_start_us":4172,"threads_started":1}
I20260929 01:01:18.461148 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.435283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.3.21.195:37327)
0929 01:01:18.448165 (+ 12882us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.448176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.451635 (+ 3459us) server_negotiation.cc:407] Connection header received
0929 01:01:18.451757 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.451768 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.452101 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.452328 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:18.454595 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.455877 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.458201 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.459715 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.459847 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.460112 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.460620 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.460813 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12651,"thread_start_us":12476,"threads_started":1}
I20260929 01:01:18.461988 10227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e945681d782472692cd317015680b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:18.464965 10516 tablet_service.cc:1467] Processing CreateTablet for tablet 4e945681d782472692cd317015680b30 (DEFAULT_TABLE table=test-table [id=3d81540f09ef423688765fc2e3194ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:18.466639 10516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e945681d782472692cd317015680b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:18.470201 10375 tablet_service.cc:1467] Processing CreateTablet for tablet 4e945681d782472692cd317015680b30 (DEFAULT_TABLE table=test-table [id=3d81540f09ef423688765fc2e3194ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:18.471757 10375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e945681d782472692cd317015680b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:18.475785 10617 tablet_bootstrap.cc:492] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Bootstrap starting.
I20260929 01:01:18.479835 10617 tablet_bootstrap.cc:654] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:18.481405 10617 log.cc:824] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:18.484588 10619 tablet_bootstrap.cc:492] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Bootstrap starting.
I20260929 01:01:18.484678 10617 tablet_bootstrap.cc:492] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: No bootstrap required, opened a new log
I20260929 01:01:18.485847 10617 ts_tablet_manager.cc:1397] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 01:01:18.488196 10620 tablet_bootstrap.cc:492] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Bootstrap starting.
I20260929 01:01:18.491185 10619 tablet_bootstrap.cc:654] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:18.492363 10620 tablet_bootstrap.cc:654] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:18.492556 10619 log.cc:824] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:18.493604 10620 log.cc:824] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:18.496382 10619 tablet_bootstrap.cc:492] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: No bootstrap required, opened a new log
I20260929 01:01:18.496706 10619 ts_tablet_manager.cc:1397] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 01:01:18.497771 10617 raft_consensus.cc:348] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.498260 10617 raft_consensus.cc:374] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:18.498368 10617 raft_consensus.cc:729] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b775d706489f4d0ea3084db13ee49cb9, State: Initialized, Role: FOLLOWER
I20260929 01:01:18.499701 10617 consensus_queue.cc:260] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.501470 10620 tablet_bootstrap.cc:492] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: No bootstrap required, opened a new log
I20260929 01:01:18.501782 10620 ts_tablet_manager.cc:1397] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260929 01:01:18.506045 10617 ts_tablet_manager.cc:1428] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260929 01:01:18.511459 10619 raft_consensus.cc:348] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.512079 10619 raft_consensus.cc:374] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:18.512244 10619 raft_consensus.cc:729] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315db3dbd8e94f5db25b298148d6ce54, State: Initialized, Role: FOLLOWER
I20260929 01:01:18.512470 10620 raft_consensus.cc:348] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.513034 10620 raft_consensus.cc:374] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:18.513149 10620 raft_consensus.cc:729] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b81c61cd9f2476ea4d6f3c41ad6a249, State: Initialized, Role: FOLLOWER
I20260929 01:01:18.513080 10619 consensus_queue.cc:260] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.513809 10620 consensus_queue.cc:260] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.515138 10619 ts_tablet_manager.cc:1428] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260929 01:01:18.515455 10620 ts_tablet_manager.cc:1428] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260929 01:01:18.515563 10582 heartbeater.cc:503] Master 127.3.21.254:36493 was elected leader, sending a full tablet report...
I20260929 01:01:18.516623 10441 heartbeater.cc:503] Master 127.3.21.254:36493 was elected leader, sending a full tablet report...
I20260929 01:01:18.555094 10624 raft_consensus.cc:482] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:18.555505 10624 raft_consensus.cc:504] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.557528 10624 leader_election.cc:288] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107), b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:18.565164 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.557953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:48143 (local address 127.3.21.194:44107)
0929 01:01:18.558197 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.558203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.558238 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:18.558465 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.558473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.558594 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.558695 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:18.559221 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.560132 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.562297 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.563994 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.564200 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.564275 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.564723 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.564863 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:01:18.565191 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.557775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:44107 (local address 127.3.21.195:48143)
0929 01:01:18.558072 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.558095 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.558257 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.558731 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.558736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.558774 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:18.559030 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.559045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.560303 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.560308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.562101 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.562114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.564238 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.564248 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.564523 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.564556 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.564681 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.564846 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":31}
I20260929 01:01:18.565419 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.558071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:33573 (local address 127.3.21.195:40955)
0929 01:01:18.558437 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:18.558458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:18.558641 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:18.559008 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:18.559011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:18.559033 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:18.559222 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.559233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.560639 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.560643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.562700 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:18.562715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.564426 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.564431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:18.564715 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.564749 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:01:18.564892 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 01:01:18.565051 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1}
I20260929 01:01:18.565653 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.558203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:40955 (local address 127.3.21.193:33573)
0929 01:01:18.558444 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.558449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.558630 (+ 181us) server_negotiation.cc:407] Connection header received
0929 01:01:18.558748 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.558753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.558839 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.558948 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:18.559453 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.560288 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.562916 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.564186 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.564360 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.564454 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.564933 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.565068 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:01:18.566288 10395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e945681d782472692cd317015680b30" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" is_pre_election: true
I20260929 01:01:18.566534 10247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e945681d782472692cd317015680b30" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d706489f4d0ea3084db13ee49cb9" is_pre_election: true
I20260929 01:01:18.566960 10395 raft_consensus.cc:2454] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 0.
I20260929 01:01:18.567168 10247 raft_consensus.cc:2454] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 0.
I20260929 01:01:18.567749 10470 leader_election.cc:302] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b81c61cd9f2476ea4d6f3c41ad6a249, 315db3dbd8e94f5db25b298148d6ce54; no voters:
I20260929 01:01:18.568259 10624 raft_consensus.cc:2781] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:18.568437 10624 raft_consensus.cc:482] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:18.568554 10624 raft_consensus.cc:3037] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:01:18.569090 10584 tablet.cc:2369] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:18.571240 10624 raft_consensus.cc:504] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.572005 10624 leader_election.cc:288] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 election: Requested vote from peers 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107), b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:18.572589 10395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e945681d782472692cd317015680b30" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249"
I20260929 01:01:18.572607 10247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e945681d782472692cd317015680b30" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d706489f4d0ea3084db13ee49cb9"
I20260929 01:01:18.572827 10395 raft_consensus.cc:3037] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:18.572850 10247 raft_consensus.cc:3037] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:18.576128 10395 raft_consensus.cc:2454] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 1.
I20260929 01:01:18.576148 10247 raft_consensus.cc:2454] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 1.
I20260929 01:01:18.576819 10472 leader_election.cc:302] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315db3dbd8e94f5db25b298148d6ce54, b775d706489f4d0ea3084db13ee49cb9; no voters:
I20260929 01:01:18.577186 10624 raft_consensus.cc:2781] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:18.577688 10624 raft_consensus.cc:686] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 LEADER]: Becoming Leader. State: Replica: 315db3dbd8e94f5db25b298148d6ce54, State: Running, Role: LEADER
I20260929 01:01:18.578795 10624 consensus_queue.cc:237] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } }
I20260929 01:01:18.584995 9322 catalog_manager.cc:5495] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195). New cstate: current_term: 1 leader_uuid: "315db3dbd8e94f5db25b298148d6ce54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } health_report { overall_health: UNKNOWN } } }
W20260929 01:01:18.586812 10443 tablet.cc:2369] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:18.616365 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.608208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.3.21.195:37327)
0929 01:01:18.608416 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.608422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.608448 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:01:18.608699 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.608703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.608802 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.608990 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:18.609808 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.610741 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.612961 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.613594 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.613722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.614386 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:18.615211 (+ 825us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:18.615346 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.615759 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.615928 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:01:18.627832 10247 raft_consensus.cc:1261] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Refusing update from remote peer 315db3dbd8e94f5db25b298148d6ce54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:18.627832 10395 raft_consensus.cc:1261] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Refusing update from remote peer 315db3dbd8e94f5db25b298148d6ce54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:18.629129 10626 consensus_queue.cc:1035] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:18.629712 10624 consensus_queue.cc:1035] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:01:18.631356 10295 tablet.cc:2369] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:18.645637 10631 mvcc.cc:205] Tried to move back new op lower bound from 7334476507644567552 to 7334476507463229440. Current Snapshot: MvccSnapshot[applied={T|T < 7334476507644567552}]
I20260929 01:01:18.667912 9321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:18.668247 9321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:18.676151 9321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59123c9daf244e67983512edc364a35a: Sending DeleteTablet for 3 replicas of tablet 4e945681d782472692cd317015680b30
I20260929 01:01:18.677459 10375 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e945681d782472692cd317015680b30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:18 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:45400
I20260929 01:01:18.677744 10516 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e945681d782472692cd317015680b30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:18 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:43046
I20260929 01:01:18.678329 10227 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e945681d782472692cd317015680b30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:18 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:43996
I20260929 01:01:18.691826 10640 tablet_replica.cc:330] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: stopping tablet replica
I20260929 01:01:18.692517 10640 raft_consensus.cc:2229] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:18.693163 10640 raft_consensus.cc:2258] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:18.695813 10640 ts_tablet_manager.cc:1905] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:18.704114 10640 ts_tablet_manager.cc:1918] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:18.704425 10640 log.cc:1196] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/4e945681d782472692cd317015680b30
I20260929 01:01:18.705372 10640 ts_tablet_manager.cc:1939] T 4e945681d782472692cd317015680b30 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Deleting consensus metadata
I20260929 01:01:18.706423 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9291
I20260929 01:01:18.707882 9353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:18.707856 9308 catalog_manager.cc:4841] TS 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107): tablet 4e945681d782472692cd317015680b30 (table test-table [id=3d81540f09ef423688765fc2e3194ad5]) successfully deleted
I20260929 01:01:18.708415 10642 tablet_replica.cc:330] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: stopping tablet replica
I20260929 01:01:18.711601 10642 raft_consensus.cc:2229] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:18.714120 10642 raft_consensus.cc:2258] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:18.716997 10642 ts_tablet_manager.cc:1905] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:18.723667 10641 tablet_replica.cc:330] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: stopping tablet replica
I20260929 01:01:18.724311 10641 raft_consensus.cc:2229] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:01:18.725148 10641 raft_consensus.cc:2258] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:18.726727 10642 ts_tablet_manager.cc:1918] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:18.726907 10642 log.cc:1196] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/4e945681d782472692cd317015680b30
I20260929 01:01:18.727463 10642 ts_tablet_manager.cc:1939] T 4e945681d782472692cd317015680b30 P b775d706489f4d0ea3084db13ee49cb9: Deleting consensus metadata
I20260929 01:01:18.728219 10641 ts_tablet_manager.cc:1905] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:18.739189 10641 ts_tablet_manager.cc:1918] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:18.739427 10641 log.cc:1196] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/4e945681d782472692cd317015680b30
I20260929 01:01:18.740259 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.685536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.3.21.253:33935)
0929 01:01:18.685733 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.685740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.704629 (+ 18889us) server_negotiation.cc:407] Connection header received
0929 01:01:18.708234 (+ 3605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.708240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.708332 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.708464 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.713632 (+ 5168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.714970 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.717118 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.717838 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.717953 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.731536 (+ 13583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.731545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.731550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.731603 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.735963 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.737083 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.737089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.737093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.737260 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.737726 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.737731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.737735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.738140 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.738403 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.739615 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.739864 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:01:18.740545 10641 ts_tablet_manager.cc:1939] T 4e945681d782472692cd317015680b30 P 315db3dbd8e94f5db25b298148d6ce54: Deleting consensus metadata
I20260929 01:01:18.761457 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.705470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.3.21.252:37453)
0929 01:01:18.705688 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.705696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.742037 (+ 36341us) server_negotiation.cc:407] Connection header received
0929 01:01:18.742351 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.742357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.742474 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.742603 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.744770 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.746277 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.748456 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.749234 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.749436 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.753910 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.753917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.753923 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.753979 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.757684 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.758752 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.758759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.758763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.758963 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.759618 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.759624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.759628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.760188 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.760640 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.760830 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.761072 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:01:18.863761 9309 catalog_manager.cc:4841] TS 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195:37327): tablet 4e945681d782472692cd317015680b30 (table test-table [id=3d81540f09ef423688765fc2e3194ad5]) successfully deleted
I20260929 01:01:18.865459 9310 catalog_manager.cc:4841] TS b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573): tablet 4e945681d782472692cd317015680b30 (table test-table [id=3d81540f09ef423688765fc2e3194ad5]) successfully deleted
I20260929 01:01:18.874636 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9291
I20260929 01:01:18.935005 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.913398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.3.21.253:33935)
0929 01:01:18.913663 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.913673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.914758 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 01:01:18.915096 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.915102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.915188 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.915270 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.917554 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.918738 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.921011 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.921655 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.921782 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.927074 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.927083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.927090 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.927151 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.930984 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.932306 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.932313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.932317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.932567 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.933057 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.933062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.933065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.933442 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.933780 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.934099 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.934341 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:01:18.935395 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:18.915056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.3.21.252:37453)
0929 01:01:18.915263 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:18.915269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:18.915492 (+ 223us) server_negotiation.cc:407] Connection header received
0929 01:01:18.915715 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:18.915721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:18.915806 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:18.915980 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:18.917883 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.918691 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.921028 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:18.921465 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:18.921546 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:18.927072 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:18.927082 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:18.927086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:18.927144 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:18.931512 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.932503 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.932513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.932519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.932859 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:18.933327 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:18.933334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:18.933338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:18.933825 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:18.934094 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:18.934770 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 01:01:18.935030 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:01:19.658463 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.657221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:41589)
0929 01:01:19.658059 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.658203 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":534,"threads_started":1}
W20260929 01:01:19.659027 10441 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:36493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
I20260929 01:01:19.659073 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.658154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:48247)
0929 01:01:19.658667 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.658836 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":241,"threads_started":1}
W20260929 01:01:19.659525 10582 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:36493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
I20260929 01:01:19.661978 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:19.661016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.193:54407)
0929 01:01:19.661604 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:19.661746 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":394,"threads_started":1}
W20260929 01:01:19.662384 10293 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:36493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
I20260929 01:01:20.176334 10654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59123c9daf244e67983512edc364a35a)
I20260929 01:01:20.176672 10655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59123c9daf244e67983512edc364a35a)
I20260929 01:01:20.176676 10654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } }
I20260929 01:01:20.176959 10655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } }
I20260929 01:01:20.178319 10654 leader_election.cc:288] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59123c9daf244e67983512edc364a35a (127.3.21.254:36493), 16375c11274e45969111ed5f425adc24 (127.3.21.252:37453)
I20260929 01:01:20.178363 10655 leader_election.cc:288] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59123c9daf244e67983512edc364a35a (127.3.21.254:36493), 65e52fc8b85e45c4a7348d71e6f3911a (127.3.21.253:33935)
I20260929 01:01:20.184656 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.178646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58962)
0929 01:01:20.179645 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.179892 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":668,"threads_started":1}
I20260929 01:01:20.186272 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.178976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58970)
0929 01:01:20.185277 (+ 6301us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.185857 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6158,"thread_start_us":6009,"threads_started":1}
W20260929 01:01:20.186741 9909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
W20260929 01:01:20.187053 10098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
I20260929 01:01:20.190686 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.190173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58998)
0929 01:01:20.190387 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.190473 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 01:01:20.190721 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.189896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:58996)
0929 01:01:20.190201 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.190417 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260929 01:01:20.190966 10098 leader_election.cc:334] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59123c9daf244e67983512edc364a35a (127.3.21.254:36493): Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
W20260929 01:01:20.191087 9909 leader_election.cc:334] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59123c9daf244e67983512edc364a35a (127.3.21.254:36493): Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
I20260929 01:01:20.195297 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.179964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.3.21.253:33935)
0929 01:01:20.186872 (+ 6908us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.186879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.186902 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:01:20.186971 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.186977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.187071 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.187264 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:20.188060 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.190340 (+ 2280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.192699 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.193983 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.194100 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.194188 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.194917 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.195031 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":6744,"thread_start_us":6521,"threads_started":1}
I20260929 01:01:20.195353 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.179901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:33935 (local address 127.0.0.1:47368)
0929 01:01:20.185111 (+ 5210us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.185140 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:20.185292 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:20.187476 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:20.187485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:20.187550 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:20.187819 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.187832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.190570 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.190576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.192516 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.192527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.194171 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.194177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.194547 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.194595 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 01:01:20.194839 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 01:01:20.195015 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5091,"thread_start_us":5665,"threads_started":1}
I20260929 01:01:20.196125 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16375c11274e45969111ed5f425adc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" is_pre_election: true
I20260929 01:01:20.196575 9934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16375c11274e45969111ed5f425adc24 in term 1.
I20260929 01:01:20.196918 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.179693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.3.21.252:37453)
0929 01:01:20.187312 (+ 7619us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.187320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.187347 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:01:20.187410 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.187415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.187511 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.188649 (+ 1138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:20.189664 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.191655 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.194144 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.195785 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.195997 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.196104 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.196361 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.196509 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7484,"thread_start_us":7093,"threads_started":1}
I20260929 01:01:20.197068 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.178588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:37453 (local address 127.0.0.1:50724)
0929 01:01:20.179327 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.179355 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:20.179523 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:20.187757 (+ 8234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:20.187766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:20.187808 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:20.188595 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.188614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.191931 (+ 3317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.191935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.193922 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.193940 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.196045 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.196050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.196199 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.196223 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:01:20.196303 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:01:20.196512 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":29,"thread_start_us":383,"threads_started":1}
I20260929 01:01:20.197403 10099 leader_election.cc:302] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16375c11274e45969111ed5f425adc24, 65e52fc8b85e45c4a7348d71e6f3911a; no voters: 59123c9daf244e67983512edc364a35a
I20260929 01:01:20.197928 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:01:20.197868 10124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16375c11274e45969111ed5f425adc24" is_pre_election: true
I20260929 01:01:20.198077 10655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59123c9daf244e67983512edc364a35a)
I20260929 01:01:20.198201 10655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:01:20.200510 10655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } }
I20260929 01:01:20.200994 10124 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65e52fc8b85e45c4a7348d71e6f3911a in current term 2: Already voted for candidate 16375c11274e45969111ed5f425adc24 in this term.
I20260929 01:01:20.201337 10655 leader_election.cc:288] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [CANDIDATE]: Term 2 election: Requested vote from peers 59123c9daf244e67983512edc364a35a (127.3.21.254:36493), 65e52fc8b85e45c4a7348d71e6f3911a (127.3.21.253:33935)
I20260929 01:01:20.201900 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16375c11274e45969111ed5f425adc24" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "65e52fc8b85e45c4a7348d71e6f3911a"
I20260929 01:01:20.202135 9934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:01:20.202174 9910 leader_election.cc:302] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65e52fc8b85e45c4a7348d71e6f3911a; no voters: 16375c11274e45969111ed5f425adc24, 59123c9daf244e67983512edc364a35a
I20260929 01:01:20.202728 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.201474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:59004)
0929 01:01:20.201692 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.202545 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 01:01:20.203944 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.203332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:59018)
0929 01:01:20.203624 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.203715 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260929 01:01:20.204237 10098 leader_election.cc:334] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 59123c9daf244e67983512edc364a35a (127.3.21.254:36493): Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
I20260929 01:01:20.205093 9934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16375c11274e45969111ed5f425adc24 in term 2.
I20260929 01:01:20.205272 10654 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:01:20.205703 10099 leader_election.cc:302] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16375c11274e45969111ed5f425adc24, 65e52fc8b85e45c4a7348d71e6f3911a; no voters: 59123c9daf244e67983512edc364a35a
I20260929 01:01:20.206070 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:01:20.206883 10655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [term 2 LEADER]: Becoming Leader. State: Replica: 16375c11274e45969111ed5f425adc24, State: Running, Role: LEADER
I20260929 01:01:20.207628 10655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } }
I20260929 01:01:20.210005 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16375c11274e45969111ed5f425adc24. Latest consensus state: current_term: 2 leader_uuid: "16375c11274e45969111ed5f425adc24" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:20.210462 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:20.211278 10666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:20.215197 10666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:20.216053 10666 catalog_manager.cc:1259] Loaded cluster ID: f310d5b134134361914cb13b4b9ab2cf
I20260929 01:01:20.216162 10666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:20.217301 10666 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:20.218151 10666 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Loaded TSK: 0
I20260929 01:01:20.219095 10666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:20.245990 10114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:01:20.254124 10114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:20.258906 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.258298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:59036)
0929 01:01:20.258646 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.258723 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260929 01:01:20.258886 9934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [term 2 FOLLOWER]: Refusing update from remote peer 16375c11274e45969111ed5f425adc24: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 01:01:20.260255 10664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 01:01:20.260525 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.259961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:59046)
0929 01:01:20.260164 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.260264 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 01:01:20.260803 10098 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 -> Peer 59123c9daf244e67983512edc364a35a (127.3.21.254:36493): Couldn't send request to peer 59123c9daf244e67983512edc364a35a. Status: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:01:20.264046 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16375c11274e45969111ed5f425adc24. Latest consensus state: current_term: 2 leader_uuid: "16375c11274e45969111ed5f425adc24" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:20.264539 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:20.266515 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "16375c11274e45969111ed5f425adc24" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:20.266984 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:20.268857 10668 mvcc.cc:205] Tried to move back new op lower bound from 7334476514332602368 to 7334476514134323200. Current Snapshot: MvccSnapshot[applied={T|T < 7334476514332602368}]
I20260929 01:01:20.269595 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "16375c11274e45969111ed5f425adc24" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "59123c9daf244e67983512edc364a35a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36493 } } peers { permanent_uuid: "65e52fc8b85e45c4a7348d71e6f3911a" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 33935 } attrs { promote: false } } peers { permanent_uuid: "16375c11274e45969111ed5f425adc24" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 37453 } attrs { promote: false } } }
I20260929 01:01:20.269876 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e52fc8b85e45c4a7348d71e6f3911a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:01:20.303510 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.280569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:33573 (local address 127.0.0.1:44010)
0929 01:01:20.282447 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.282475 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:20.282668 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:20.294924 (+ 12256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:20.294931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:20.295099 (+ 168us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:20.295388 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.295402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.297626 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.297639 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.299747 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.299760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.302109 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.302116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.302456 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.302712 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 01:01:20.302925 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 01:01:20.303119 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1678,"mutex_wait_us":82}
I20260929 01:01:20.303828 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.280964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.3.21.193:33573)
0929 01:01:20.293975 (+ 13011us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.293984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.294031 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:01:20.294180 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.294187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.294306 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.295604 (+ 1298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:20.295820 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.296958 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.300079 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.301850 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.302100 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.302267 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.302983 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.303164 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12679,"thread_start_us":13680,"threads_started":1}
I20260929 01:01:20.304796 10227 tablet_service.cc:1467] Processing CreateTablet for tablet d4332aca0fc34328ad745107183fb970 (DEFAULT_TABLE table=test-table [id=f9947882e01c4169b5c9f44ccaf0ee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:20.305300 10227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4332aca0fc34328ad745107183fb970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:20.306450 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.280372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.3.21.194:44107)
0929 01:01:20.296859 (+ 16487us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.296867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.296913 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:01:20.297006 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.297011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.297100 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.297727 (+ 627us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:20.298183 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.299281 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.304008 (+ 4727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.305040 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.305231 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.305355 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.305983 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.306149 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16343,"thread_start_us":16883,"threads_started":1}
I20260929 01:01:20.306941 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.280439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:44107 (local address 127.0.0.1:45410)
0929 01:01:20.280731 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.280753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:20.280886 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:20.297458 (+ 16572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:20.297465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:20.297511 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:20.297991 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.298004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.299462 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.299468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.303577 (+ 4109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.303596 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.305329 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.305338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.305582 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.305621 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:01:20.305943 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 01:01:20.306290 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":259}
I20260929 01:01:20.309514 10375 tablet_service.cc:1467] Processing CreateTablet for tablet d4332aca0fc34328ad745107183fb970 (DEFAULT_TABLE table=test-table [id=f9947882e01c4169b5c9f44ccaf0ee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:20.310004 10375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4332aca0fc34328ad745107183fb970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:20.310382 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.281483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.3.21.195:37327)
0929 01:01:20.296859 (+ 15376us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.296867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.296913 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:01:20.297006 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.297011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.297100 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.297418 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:20.298806 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.299786 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.307573 (+ 7787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.308515 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.308711 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.308861 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.309861 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.310106 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15197,"thread_start_us":16095,"threads_started":1}
I20260929 01:01:20.310874 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.281320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37327 (local address 127.0.0.1:43076)
0929 01:01:20.288853 (+ 7533us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.288895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:20.289128 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:20.297823 (+ 8695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:20.297828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:20.297936 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:20.298603 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.298616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.300029 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.300035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.307327 (+ 7292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:20.307347 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.308824 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.308834 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:20.309054 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.309099 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:01:20.309192 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 01:01:20.309359 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7306,"mutex_wait_us":94,"thread_start_us":7935,"threads_started":1}
I20260929 01:01:20.311932 10516 tablet_service.cc:1467] Processing CreateTablet for tablet d4332aca0fc34328ad745107183fb970 (DEFAULT_TABLE table=test-table [id=f9947882e01c4169b5c9f44ccaf0ee39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:20.312520 10516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4332aca0fc34328ad745107183fb970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:20.320758 10677 tablet_bootstrap.cc:492] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Bootstrap starting.
I20260929 01:01:20.325138 10677 tablet_bootstrap.cc:654] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:20.327261 10678 tablet_bootstrap.cc:492] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Bootstrap starting.
I20260929 01:01:20.327898 10679 tablet_bootstrap.cc:492] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Bootstrap starting.
I20260929 01:01:20.328917 10678 tablet_bootstrap.cc:654] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:20.329604 10679 tablet_bootstrap.cc:654] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:20.331285 10677 tablet_bootstrap.cc:492] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: No bootstrap required, opened a new log
I20260929 01:01:20.331724 10677 ts_tablet_manager.cc:1397] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 01:01:20.332610 10677 raft_consensus.cc:348] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.332876 10677 raft_consensus.cc:374] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:20.333007 10677 raft_consensus.cc:729] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b775d706489f4d0ea3084db13ee49cb9, State: Initialized, Role: FOLLOWER
I20260929 01:01:20.333295 10677 consensus_queue.cc:260] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.333572 10679 tablet_bootstrap.cc:492] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: No bootstrap required, opened a new log
I20260929 01:01:20.333771 10679 ts_tablet_manager.cc:1397] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:01:20.334753 10677 ts_tablet_manager.cc:1428] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260929 01:01:20.335180 10678 tablet_bootstrap.cc:492] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: No bootstrap required, opened a new log
I20260929 01:01:20.335361 10678 ts_tablet_manager.cc:1397] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 01:01:20.336380 10678 raft_consensus.cc:348] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.336727 10678 raft_consensus.cc:374] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:20.336879 10678 raft_consensus.cc:729] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b81c61cd9f2476ea4d6f3c41ad6a249, State: Initialized, Role: FOLLOWER
I20260929 01:01:20.337242 10678 consensus_queue.cc:260] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.338706 10292 heartbeater.cc:503] Master 127.3.21.252:37453 was elected leader, sending a full tablet report...
I20260929 01:01:20.338886 10678 ts_tablet_manager.cc:1428] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:20.341756 10440 heartbeater.cc:503] Master 127.3.21.252:37453 was elected leader, sending a full tablet report...
I20260929 01:01:20.343814 10679 raft_consensus.cc:348] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.344224 10679 raft_consensus.cc:374] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:20.344417 10679 raft_consensus.cc:729] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315db3dbd8e94f5db25b298148d6ce54, State: Initialized, Role: FOLLOWER
I20260929 01:01:20.344736 10679 consensus_queue.cc:260] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.347512 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.335699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.193:34905)
0929 01:01:20.347105 (+ 11406us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.347268 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11222,"thread_start_us":11223,"threads_started":1}
I20260929 01:01:20.347595 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.339518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:38793)
0929 01:01:20.347080 (+ 7562us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.347280 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7417,"thread_start_us":7200,"threads_started":1}
I20260929 01:01:20.348953 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.348429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:54013)
0929 01:01:20.348637 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.348754 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 01:01:20.350668 10679 ts_tablet_manager.cc:1428] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Time spent starting tablet: real 0.017s user 0.001s sys 0.006s
I20260929 01:01:20.353222 10581 heartbeater.cc:503] Master 127.3.21.252:37453 was elected leader, sending a full tablet report...
I20260929 01:01:20.353233 10687 raft_consensus.cc:482] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:20.353533 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.351657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:53151)
0929 01:01:20.352555 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.352839 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":548,"threads_started":1}
I20260929 01:01:20.353578 10687 raft_consensus.cc:504] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.355146 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.354604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:58381)
0929 01:01:20.354842 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.354967 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 01:01:20.355688 10395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4332aca0fc34328ad745107183fb970" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" is_pre_election: true
I20260929 01:01:20.355847 10247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4332aca0fc34328ad745107183fb970" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d706489f4d0ea3084db13ee49cb9" is_pre_election: true
I20260929 01:01:20.355993 10395 raft_consensus.cc:2454] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 0.
I20260929 01:01:20.356099 10247 raft_consensus.cc:2454] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 0.
I20260929 01:01:20.356613 10472 leader_election.cc:302] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315db3dbd8e94f5db25b298148d6ce54, b775d706489f4d0ea3084db13ee49cb9; no voters:
I20260929 01:01:20.357316 10687 leader_election.cc:288] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107), b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:20.357389 10689 raft_consensus.cc:2781] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:20.357700 10689 raft_consensus.cc:482] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:20.357843 10689 raft_consensus.cc:3037] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:20.360133 10689 raft_consensus.cc:504] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.360947 10689 leader_election.cc:288] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 election: Requested vote from peers 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107), b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573)
I20260929 01:01:20.361479 10395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4332aca0fc34328ad745107183fb970" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249"
I20260929 01:01:20.361474 10247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4332aca0fc34328ad745107183fb970" candidate_uuid: "315db3dbd8e94f5db25b298148d6ce54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d706489f4d0ea3084db13ee49cb9"
I20260929 01:01:20.361739 10247 raft_consensus.cc:3037] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:20.361745 10395 raft_consensus.cc:3037] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:20.363942 10395 raft_consensus.cc:2454] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 1.
I20260929 01:01:20.363987 10247 raft_consensus.cc:2454] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315db3dbd8e94f5db25b298148d6ce54 in term 1.
I20260929 01:01:20.364506 10470 leader_election.cc:302] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b81c61cd9f2476ea4d6f3c41ad6a249, 315db3dbd8e94f5db25b298148d6ce54; no voters:
I20260929 01:01:20.364858 10689 raft_consensus.cc:2781] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:20.365012 10689 raft_consensus.cc:686] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 LEADER]: Becoming Leader. State: Replica: 315db3dbd8e94f5db25b298148d6ce54, State: Running, Role: LEADER
I20260929 01:01:20.365456 10689 consensus_queue.cc:237] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } }
I20260929 01:01:20.366506 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.365948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:33903)
0929 01:01:20.366151 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.366291 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 01:01:20.369971 10114 catalog_manager.cc:5495] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195). New cstate: current_term: 1 leader_uuid: "315db3dbd8e94f5db25b298148d6ce54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315db3dbd8e94f5db25b298148d6ce54" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37327 } health_report { overall_health: HEALTHY } } }
I20260929 01:01:20.402180 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.394924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.3.21.195:37327)
0929 01:01:20.395136 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.395140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.395163 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:01:20.395439 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.395443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.395513 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.395602 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:20.396090 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.396997 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.398823 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.399612 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.399834 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.400948 (+ 1114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:20.401115 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:20.401240 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.401684 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.401785 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:01:20.407630 10247 raft_consensus.cc:1261] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Refusing update from remote peer 315db3dbd8e94f5db25b298148d6ce54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:20.407645 10395 raft_consensus.cc:1261] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Refusing update from remote peer 315db3dbd8e94f5db25b298148d6ce54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:20.408469 10687 consensus_queue.cc:1035] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b775d706489f4d0ea3084db13ee49cb9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:20.408846 10689 consensus_queue.cc:1035] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b81c61cd9f2476ea4d6f3c41ad6a249" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:20.412001 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.411491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:38883)
0929 01:01:20.411709 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.411832 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 01:01:20.412279 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.411181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.193:56167)
0929 01:01:20.411454 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.411558 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260929 01:01:20.418946 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.417477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.194:51105)
0929 01:01:20.418555 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.418699 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":857}
I20260929 01:01:20.423183 10693 mvcc.cc:205] Tried to move back new op lower bound from 7334476514940575744 to 7334476514779729920. Current Snapshot: MvccSnapshot[applied={T|T < 7334476514940575744}]
I20260929 01:01:20.428144 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.427614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.193:33679)
0929 01:01:20.427801 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.427935 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 01:01:20.432792 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.432206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:35737)
0929 01:01:20.432464 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.432573 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 01:01:20.434801 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.434211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.3.21.195:41667)
0929 01:01:20.434476 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.434594 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260929 01:01:20.440460 10114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:20.440763 10114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:20.449137 10114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16375c11274e45969111ed5f425adc24: Sending DeleteTablet for 3 replicas of tablet d4332aca0fc34328ad745107183fb970
I20260929 01:01:20.450611 10375 tablet_service.cc:1514] Processing DeleteTablet for tablet d4332aca0fc34328ad745107183fb970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:20 UTC) from {username='oryx', principal='oryx/127.3.21.252@KRBTEST.COM'} at 127.0.0.1:45410
I20260929 01:01:20.450850 10227 tablet_service.cc:1514] Processing DeleteTablet for tablet d4332aca0fc34328ad745107183fb970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:20 UTC) from {username='oryx', principal='oryx/127.3.21.252@KRBTEST.COM'} at 127.0.0.1:44010
I20260929 01:01:20.451475 10516 tablet_service.cc:1514] Processing DeleteTablet for tablet d4332aca0fc34328ad745107183fb970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:20 UTC) from {username='oryx', principal='oryx/127.3.21.252@KRBTEST.COM'} at 127.0.0.1:43076
I20260929 01:01:20.453634 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 10101
I20260929 01:01:20.462581 10701 tablet_replica.cc:330] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: stopping tablet replica
I20260929 01:01:20.463034 10701 raft_consensus.cc:2229] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:20.463574 10701 raft_consensus.cc:2258] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:20.463771 10703 tablet_replica.cc:330] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: stopping tablet replica
I20260929 01:01:20.464097 10703 raft_consensus.cc:2229] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:01:20.464605 10703 raft_consensus.cc:2258] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:20.465350 10702 tablet_replica.cc:330] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: stopping tablet replica
I20260929 01:01:20.465770 10701 ts_tablet_manager.cc:1905] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:20.465970 10703 ts_tablet_manager.cc:1905] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:20.466506 10702 raft_consensus.cc:2229] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:20.468621 10702 raft_consensus.cc:2258] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:20.469795 10702 ts_tablet_manager.cc:1905] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:20.473889 10701 ts_tablet_manager.cc:1918] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:20.474032 10703 ts_tablet_manager.cc:1918] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:20.474079 10701 log.cc:1196] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/d4332aca0fc34328ad745107183fb970
I20260929 01:01:20.474170 10703 log.cc:1196] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/d4332aca0fc34328ad745107183fb970
I20260929 01:01:20.474670 10703 ts_tablet_manager.cc:1939] T d4332aca0fc34328ad745107183fb970 P 315db3dbd8e94f5db25b298148d6ce54: Deleting consensus metadata
I20260929 01:01:20.474798 10701 ts_tablet_manager.cc:1939] T d4332aca0fc34328ad745107183fb970 P 1b81c61cd9f2476ea4d6f3c41ad6a249: Deleting consensus metadata
I20260929 01:01:20.476152 10098 catalog_manager.cc:4841] TS 1b81c61cd9f2476ea4d6f3c41ad6a249 (127.3.21.194:44107): tablet d4332aca0fc34328ad745107183fb970 (table test-table [id=f9947882e01c4169b5c9f44ccaf0ee39]) successfully deleted
I20260929 01:01:20.476128 10099 catalog_manager.cc:4841] TS 315db3dbd8e94f5db25b298148d6ce54 (127.3.21.195:37327): tablet d4332aca0fc34328ad745107183fb970 (table test-table [id=f9947882e01c4169b5c9f44ccaf0ee39]) successfully deleted
I20260929 01:01:20.477260 10702 ts_tablet_manager.cc:1918] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:20.477432 10702 log.cc:1196] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/d4332aca0fc34328ad745107183fb970
I20260929 01:01:20.477866 10702 ts_tablet_manager.cc:1939] T d4332aca0fc34328ad745107183fb970 P b775d706489f4d0ea3084db13ee49cb9: Deleting consensus metadata
I20260929 01:01:20.479537 10100 catalog_manager.cc:4841] TS b775d706489f4d0ea3084db13ee49cb9 (127.3.21.193:33573): tablet d4332aca0fc34328ad745107183fb970 (table test-table [id=f9947882e01c4169b5c9f44ccaf0ee39]) successfully deleted
I20260929 01:01:20.487282 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.454694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.3.21.193:33573)
0929 01:01:20.455037 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.455047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.466018 (+ 10971us) server_negotiation.cc:407] Connection header received
0929 01:01:20.466347 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.466353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.466467 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.466607 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.468350 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.469647 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.472350 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.473088 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.473200 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.477579 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:20.477596 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:20.477617 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:20.477657 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:20.481964 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:20.483475 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:20.483486 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:20.483489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:20.483796 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:20.484415 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:20.484421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:20.484424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:20.485232 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:20.485631 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.486284 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.486715 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 01:01:20.488471 10287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:20.609520 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 10101
I20260929 01:01:20.629451 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 10304
I20260929 01:01:20.646853 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.630874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.3.21.194:44107)
0929 01:01:20.631118 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.631124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.631147 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:01:20.631267 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.631272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.631358 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.631480 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.632972 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.633854 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.635538 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.636077 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.636252 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.639350 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:20.639373 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:20.639391 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:20.639432 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:20.642924 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:20.643882 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:20.643893 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:20.643896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:20.644048 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:20.644752 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:20.644757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:20.644761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:20.645190 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:20.645549 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.646053 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.646365 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:01:20.647805 10435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:20.754251 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 10304
I20260929 01:01:20.776121 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 10453
I20260929 01:01:20.794886 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.777351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.3.21.195:37327)
0929 01:01:20.777560 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:20.777564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:20.777587 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:01:20.777647 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:20.777650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:20.777712 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:20.777809 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:20.779668 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.780861 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.782732 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:20.783315 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:20.783468 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:20.786780 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:20.786801 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:20.786824 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:20.786873 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:20.791127 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:20.792133 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:20.792142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:20.792144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:20.792322 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:20.792987 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:20.792992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:20.792994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:20.793411 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:20.793678 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:20.794084 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 01:01:20.794459 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:01:20.795845 10576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:20.820852 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.819937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:59060)
0929 01:01:20.820532 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.820631 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":383,"threads_started":1}
I20260929 01:01:20.823308 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:20.822665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36493 (local address 127.0.0.1:59076)
0929 01:01:20.822867 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:20.822998 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:01:20.937793 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 10453
I20260929 01:01:20.963399 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9787
I20260929 01:01:20.964744 9954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:21.096014 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9787
I20260929 01:01:21.120537 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 9887
I20260929 01:01:21.121897 10144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:21.257786 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 9887
2026-09-29T01:01:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10469 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:21 dist-test-slave-k0b7 krb5kdc[10714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:21 dist-test-slave-k0b7 krb5kdc[10714](info): set up 2 sockets
Sep 29 01:01:21 dist-test-slave-k0b7 krb5kdc[10714](info): commencing operation
krb5kdc: starting...
W20260929 01:01:22.444224 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:01:22 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab.
Entry for principal oryx/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab.
I20260929 01:01:22.691717 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:38723
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab
--principal=oryx/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:38723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:23.051354 10727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:23.051729 10727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:23.051903 10727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:23.059783 10727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:23.059950 10727 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:01:23.060057 10727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:23.060166 10727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:23.060277 10727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:23.060369 10727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:23.071264 10727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:38723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.254.keytab
--principal=oryx/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:38723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:23.072764 10727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:23.074213 10727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:23.088023 10732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:23.088003 10735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:23.088052 10733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:23.089210 10727 server_base.cc:1034] running on GCE node
I20260929 01:01:23.090165 10727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:23.091907 10727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:23.093200 10727 hybrid_clock.cc:648] HybridClock initialized: now 1790643683093117 us; error 57 us; skew 500 ppm
I20260929 01:01:23.093585 10727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:23 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643683, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:23.097465 10727 init.cc:377] Logged in from keytab as oryx/127.3.21.254@KRBTEST.COM (short username oryx)
I20260929 01:01:23.099519 10727 webserver.cc:460] Webserver started at http://127.3.21.254:46727/ using document root <none> and password file <none>
I20260929 01:01:23.100327 10727 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:23.100473 10727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:23.100862 10727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:23.104154 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "579109f95409468784e43f7545c9acc3"
format_stamp: "Formatted at 2026-09-29 01:01:23 on dist-test-slave-k0b7"
server_key: "0d3dc78c0e14742f64772eaa312ea1c2"
server_key_iv: "b4b99678b29ff01606abb12a0321ea63"
server_key_version: "encryptionkey@0"
I20260929 01:01:23.105054 10727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "579109f95409468784e43f7545c9acc3"
format_stamp: "Formatted at 2026-09-29 01:01:23 on dist-test-slave-k0b7"
server_key: "0d3dc78c0e14742f64772eaa312ea1c2"
server_key_iv: "b4b99678b29ff01606abb12a0321ea63"
server_key_version: "encryptionkey@0"
I20260929 01:01:23.111632 10727 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 01:01:23.116592 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:23.118716 10727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:01:23.118955 10727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "579109f95409468784e43f7545c9acc3"
format_stamp: "Formatted at 2026-09-29 01:01:23 on dist-test-slave-k0b7"
server_key: "0d3dc78c0e14742f64772eaa312ea1c2"
server_key_iv: "b4b99678b29ff01606abb12a0321ea63"
server_key_version: "encryptionkey@0"
I20260929 01:01:23.119217 10727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:23.141819 10727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:23.146633 10727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:23.146931 10727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:23.170608 10727 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:38723
I20260929 01:01:23.170676 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:38723 every 8 connection(s)
I20260929 01:01:23.172070 10727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:23.176578 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:23.177167 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 10727
I20260929 01:01:23.177389 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:23.178297 3159 external_mini_cluster.cc:1428] Setting key 2717eda6243e5e054e5d04801b048be8
I20260929 01:01:23.193378 10804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Bootstrap starting.
Sep 29 01:01:23 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:23.197590 10804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:23.199090 10804 log.cc:824] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:23.202651 10804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: No bootstrap required, opened a new log
I20260929 01:01:23.205153 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:23.181030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.3.21.254:38723)
0929 01:01:23.181996 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.182008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.182053 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:01:23.184099 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.184129 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.184645 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.185067 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.187348 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.188795 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.190814 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.191641 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.191792 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:23.197027 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:23.197037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:23.197054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:23.197098 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:23.200637 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:23.201500 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:23.201506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:23.201513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:23.201796 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:23.202197 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:23.202204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:23.202211 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:23.202878 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:23.203219 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:23.203636 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 01:01:23.204098 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":469,"threads_started":1}
I20260929 01:01:23.211635 10804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579109f95409468784e43f7545c9acc3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38723 } }
I20260929 01:01:23.212090 10804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:23.212203 10804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 579109f95409468784e43f7545c9acc3, State: Initialized, Role: FOLLOWER
I20260929 01:01:23.212947 10804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579109f95409468784e43f7545c9acc3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38723 } }
I20260929 01:01:23.213285 10804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:23.213446 10804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:23.213666 10804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:23.216217 10804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579109f95409468784e43f7545c9acc3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38723 } }
I20260929 01:01:23.216835 10804 leader_election.cc:302] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 579109f95409468784e43f7545c9acc3; no voters:
I20260929 01:01:23.217584 10804 leader_election.cc:288] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:23.217760 10809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:23.219050 10809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [term 1 LEADER]: Becoming Leader. State: Replica: 579109f95409468784e43f7545c9acc3, State: Running, Role: LEADER
I20260929 01:01:23.219645 10809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579109f95409468784e43f7545c9acc3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38723 } }
I20260929 01:01:23.219985 10804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:23.222872 10811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 579109f95409468784e43f7545c9acc3. Latest consensus state: current_term: 1 leader_uuid: "579109f95409468784e43f7545c9acc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579109f95409468784e43f7545c9acc3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38723 } } }
I20260929 01:01:23.223394 10811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:23.223788 10810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "579109f95409468784e43f7545c9acc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579109f95409468784e43f7545c9acc3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38723 } } }
I20260929 01:01:23.224186 10810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:23.225482 10818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:23.230088 10818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:23.240629 10818 catalog_manager.cc:1347] Generated new cluster ID: f659d71e39a0461ab797830aaeb26158
I20260929 01:01:23.240767 10818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:23.263216 10818 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:23.264755 10818 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:23.282598 10818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Generated new TSK 0
I20260929 01:01:23.283731 10818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab.
Entry for principal oryx/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab.
I20260929 01:01:23.355830 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38723
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:23.709486 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:23.709795 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:23.709949 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:23.717648 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:23.717780 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:23.717926 10830 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:01:23.718084 10830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:23.729202 10830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:38723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:23.730728 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:23.732300 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:23.746989 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:23.747004 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:23.747026 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:23.748243 10830 server_base.cc:1034] running on GCE node
I20260929 01:01:23.748971 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:23.750030 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:23.751262 10830 hybrid_clock.cc:648] HybridClock initialized: now 1790643683751188 us; error 43 us; skew 500 ppm
I20260929 01:01:23.751626 10830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:23 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643683, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:23.760932 10830 init.cc:377] Logged in from keytab as oryx/127.3.21.193@KRBTEST.COM (short username oryx)
I20260929 01:01:23.762915 10830 webserver.cc:460] Webserver started at http://127.3.21.193:46481/ using document root <none> and password file <none>
I20260929 01:01:23.763685 10830 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:23.763790 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:23.764083 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:23.767573 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "cbbadfda961147e4875ab1359c520999"
format_stamp: "Formatted at 2026-09-29 01:01:23 on dist-test-slave-k0b7"
server_key: "673a6693d9fe46d9d9601964ce686d0f"
server_key_iv: "3f4dd240f957b47571ed1ee9d99b59aa"
server_key_version: "encryptionkey@0"
I20260929 01:01:23.768743 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "cbbadfda961147e4875ab1359c520999"
format_stamp: "Formatted at 2026-09-29 01:01:23 on dist-test-slave-k0b7"
server_key: "673a6693d9fe46d9d9601964ce686d0f"
server_key_iv: "3f4dd240f957b47571ed1ee9d99b59aa"
server_key_version: "encryptionkey@0"
I20260929 01:01:23.774962 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:01:23.779138 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:23.780985 10830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:23.781328 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "cbbadfda961147e4875ab1359c520999"
format_stamp: "Formatted at 2026-09-29 01:01:23 on dist-test-slave-k0b7"
server_key: "673a6693d9fe46d9d9601964ce686d0f"
server_key_iv: "3f4dd240f957b47571ed1ee9d99b59aa"
server_key_version: "encryptionkey@0"
I20260929 01:01:23.781625 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:23.797914 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:23.803249 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:23.803571 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:23.806686 10830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:23.806810 10830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:23.806983 10830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:23.807078 10830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:01:23 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643683, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:23.837670 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:23.808154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:46247 (local address 127.3.21.254:38723)
0929 01:01:23.809088 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.809095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.810693 (+ 1598us) server_negotiation.cc:407] Connection header received
0929 01:01:23.813219 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.813224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.813334 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.813469 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.816006 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.817314 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.819768 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.820631 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.820832 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:23.826684 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:23.826693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:23.826698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:23.826765 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:23.831082 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:23.832224 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:23.832230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:23.832234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:23.832456 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:23.833388 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:23.833393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:23.833395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:23.833695 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:23.833869 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:23.837056 (+ 3187us) server_negotiation.cc:299] Negotiation successful
0929 01:01:23.837320 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":580,"threads_started":1}
I20260929 01:01:23.837857 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:23.809046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.3.21.193:46247)
0929 01:01:23.810451 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:23.810531 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:23.812811 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:23.813757 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:23.813766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:23.814788 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:23.815730 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:23.815756 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.817525 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.817532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:23.819541 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:23.819561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.821193 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.821203 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:23.821525 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:23.822638 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:23.822662 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:23.826275 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:23.831323 (+ 5048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:23.831336 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:23.831366 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:23.832050 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:23.832635 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:23.832645 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:23.832651 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:23.833074 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:23.833873 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:23.833888 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:23.834606 (+ 718us) client_negotiation.cc:769] Sending connection context
0929 01:01:23.835033 (+ 427us) client_negotiation.cc:240] Negotiation successful
0929 01:01:23.835913 (+ 880us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":459,"threads_started":1}
I20260929 01:01:23.860160 10830 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:34755
I20260929 01:01:23.860172 10965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:34755 every 8 connection(s)
I20260929 01:01:23.861945 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:23.864459 10966 heartbeater.cc:347] Connected to a master server at 127.3.21.254:38723
I20260929 01:01:23.864738 10966 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:23.864868 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:23.847728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.3.21.193:34151)
0929 01:01:23.848374 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:23.848390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:23.848580 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:23.848997 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:23.849001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:23.849686 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:23.850240 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:23.850262 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.851530 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.851534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:23.853183 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:23.853235 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.854575 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.854583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:23.854775 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:23.855718 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:23.855762 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:23.857218 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:23.861518 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:23.861522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:23.861525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:23.862208 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:23.862803 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:23.862808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:23.862812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:23.863017 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:23.863731 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:23.863736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:23.863924 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 01:01:23.864075 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:01:23.864457 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":366,"threads_started":1}
I20260929 01:01:23.865607 10966 heartbeater.cc:511] Master 127.3.21.254:38723 requested a full tablet report, sending...
I20260929 01:01:23.865629 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:23.847721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:34151 (local address 127.3.21.254:38723)
0929 01:01:23.847899 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:23.847903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:23.848494 (+ 591us) server_negotiation.cc:407] Connection header received
0929 01:01:23.848736 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:23.848744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:23.848814 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:23.848920 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:23.850443 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.851358 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.853448 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:23.854329 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:23.854509 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:23.857506 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:23.857517 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:23.857525 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:23.857582 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:23.861335 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:23.862368 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:23.862378 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:23.862384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:23.862648 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:23.863164 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:23.863172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:23.863178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:23.863586 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:23.863754 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:23.865207 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0929 01:01:23.865407 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:01:23.868139 10759 ts_manager.cc:194] Registered new tserver with Master: cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755)
I20260929 01:01:23.869621 10759 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:46247
I20260929 01:01:23.870204 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 10830
I20260929 01:01:23.870416 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:23.871213 3159 external_mini_cluster.cc:1428] Setting key 4d104cb9f3d46cf3f34a334ee4424725
WARNING: no policy specified for oryx/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab.
Entry for principal oryx/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab.
I20260929 01:01:23.927807 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38723
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:24.295682 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:24.296000 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:24.296101 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:24.303845 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:24.303977 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:24.304064 10970 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:01:24.304143 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:24.315099 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:38723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:24.316470 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:24.317818 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:24.334187 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:24.334230 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:24.333997 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:24.335685 10970 server_base.cc:1034] running on GCE node
I20260929 01:01:24.336486 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:24.337519 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:24.338789 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790643684338739 us; error 53 us; skew 500 ppm
I20260929 01:01:24.339231 10970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:24 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643684, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:24.343382 10970 init.cc:377] Logged in from keytab as oryx/127.3.21.194@KRBTEST.COM (short username oryx)
I20260929 01:01:24.345554 10970 webserver.cc:460] Webserver started at http://127.3.21.194:43455/ using document root <none> and password file <none>
I20260929 01:01:24.346370 10970 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:24.346468 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:24.347098 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:24.350956 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "874188f28c5449a786ccb82684ed4e56"
format_stamp: "Formatted at 2026-09-29 01:01:24 on dist-test-slave-k0b7"
server_key: "dc67b0bc61d7a9b43927bc43a5f351f4"
server_key_iv: "cee7f412e345691b67364e1839dd56fb"
server_key_version: "encryptionkey@0"
I20260929 01:01:24.351866 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "874188f28c5449a786ccb82684ed4e56"
format_stamp: "Formatted at 2026-09-29 01:01:24 on dist-test-slave-k0b7"
server_key: "dc67b0bc61d7a9b43927bc43a5f351f4"
server_key_iv: "cee7f412e345691b67364e1839dd56fb"
server_key_version: "encryptionkey@0"
I20260929 01:01:24.358414 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:01:24.362627 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:24.364475 10970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:24.364686 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "874188f28c5449a786ccb82684ed4e56"
format_stamp: "Formatted at 2026-09-29 01:01:24 on dist-test-slave-k0b7"
server_key: "dc67b0bc61d7a9b43927bc43a5f351f4"
server_key_iv: "cee7f412e345691b67364e1839dd56fb"
server_key_version: "encryptionkey@0"
I20260929 01:01:24.364953 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:24.394953 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:24.401430 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:24.401821 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:24.405652 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:24.405778 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:24.405962 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:24.406049 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:01:24 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643684, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:24.442260 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.406390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:36531 (local address 127.3.21.254:38723)
0929 01:01:24.407115 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.407120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.409135 (+ 2015us) server_negotiation.cc:407] Connection header received
0929 01:01:24.411872 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.411879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.411986 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.412098 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:24.415460 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.417838 (+ 2378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.422829 (+ 4991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.423780 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.423977 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.432285 (+ 8308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:24.432295 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:24.432303 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:24.432373 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:24.437290 (+ 4917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:24.438735 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:24.438741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:24.438752 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:24.438893 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:24.439964 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:24.439974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:24.439977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:24.440313 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:24.440516 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.441479 (+ 963us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.441795 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":510,"threads_started":1}
I20260929 01:01:24.443450 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.407851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.3.21.194:36531)
0929 01:01:24.408915 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:24.408972 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:24.411461 (+ 2489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:24.412421 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:24.412435 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:24.413812 (+ 1377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:24.415076 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:24.415146 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.418779 (+ 3633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.418811 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:24.422461 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:24.422554 (+ 93us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.424111 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.424118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:24.424426 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.425393 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:24.425427 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:24.431844 (+ 6417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:24.437518 (+ 5674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:24.437625 (+ 107us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:24.437647 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:24.438556 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:24.439437 (+ 881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:24.439445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:24.439449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:24.439795 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:24.440493 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:24.440502 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:24.441015 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 01:01:24.441327 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 01:01:24.442019 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":509,"threads_started":1}
I20260929 01:01:24.469722 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.451033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:46953 (local address 127.3.21.254:38723)
0929 01:01:24.451238 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:24.451242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:24.451969 (+ 727us) server_negotiation.cc:407] Connection header received
0929 01:01:24.452194 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:24.452199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:24.452287 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:24.452410 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:24.454371 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.455815 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.458691 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.459360 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.459522 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.461661 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:24.461673 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:24.461677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:24.461709 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:24.465826 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:24.466848 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:24.466857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:24.466862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:24.467063 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:24.468010 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:24.468017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:24.468021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:24.468396 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:24.468571 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:24.469075 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 01:01:24.469349 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:01:24.469880 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:24.451120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.3.21.194:46953)
0929 01:01:24.451824 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:24.451842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:24.452056 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:24.452440 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:24.452445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:24.453404 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:24.454125 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:24.454158 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.456040 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.456047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:24.458404 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:24.458430 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:24.459779 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:24.459784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:24.459904 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:24.460601 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:24.460620 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:24.461450 (+ 830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:24.466021 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:24.466026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:24.466029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:24.466693 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:24.467383 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:24.467397 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:24.467402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:24.467810 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:24.468556 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:24.468561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:24.468759 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 01:01:24.468981 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 01:01:24.469349 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":424,"threads_started":1}
I20260929 01:01:24.481877 10970 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:37045
I20260929 01:01:24.481889 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:37045 every 8 connection(s)
I20260929 01:01:24.483354 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:24.484077 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 10970
I20260929 01:01:24.484264 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:24.484992 3159 external_mini_cluster.cc:1428] Setting key f64d9a964bfd839e130d96698fd97bde
I20260929 01:01:24.486339 11106 heartbeater.cc:347] Connected to a master server at 127.3.21.254:38723
I20260929 01:01:24.486708 11106 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:24.487488 11106 heartbeater.cc:511] Master 127.3.21.254:38723 requested a full tablet report, sending...
I20260929 01:01:24.489380 10759 ts_manager.cc:194] Registered new tserver with Master: 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045)
I20260929 01:01:24.490873 10759 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:36531
WARNING: no policy specified for oryx/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab.
Entry for principal oryx/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab.
I20260929 01:01:24.543283 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:38723
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:24.873384 10966 heartbeater.cc:503] Master 127.3.21.254:38723 was elected leader, sending a full tablet report...
W20260929 01:01:24.910977 11110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:24.911366 11110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:24.911535 11110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:24.919514 11110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:24.919708 11110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:24.919891 11110 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:01:24.920037 11110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:24.931437 11110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.21.254:38723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:24.932963 11110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:24.934433 11110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:24.950835 11118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:24.950876 11116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:24.950759 11115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:24.952291 11110 server_base.cc:1034] running on GCE node
I20260929 01:01:24.953958 11110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:24.955554 11110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:24.957080 11110 hybrid_clock.cc:648] HybridClock initialized: now 1790643684956980 us; error 101 us; skew 500 ppm
I20260929 01:01:24.957477 11110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:24 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643684, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:24.961553 11110 init.cc:377] Logged in from keytab as oryx/127.3.21.195@KRBTEST.COM (short username oryx)
I20260929 01:01:24.963667 11110 webserver.cc:460] Webserver started at http://127.3.21.195:35995/ using document root <none> and password file <none>
I20260929 01:01:24.964529 11110 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:24.964684 11110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:24.965101 11110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:24.969218 11110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "bac5683e28774f5cbe0621383f97f900"
format_stamp: "Formatted at 2026-09-29 01:01:24 on dist-test-slave-k0b7"
server_key: "35f09b8ec26107cc5978041498582006"
server_key_iv: "96e54ad25d2edd7851bc62363d3409df"
server_key_version: "encryptionkey@0"
I20260929 01:01:24.970247 11110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "bac5683e28774f5cbe0621383f97f900"
format_stamp: "Formatted at 2026-09-29 01:01:24 on dist-test-slave-k0b7"
server_key: "35f09b8ec26107cc5978041498582006"
server_key_iv: "96e54ad25d2edd7851bc62363d3409df"
server_key_version: "encryptionkey@0"
I20260929 01:01:24.977080 11110 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:01:24.982388 11125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:24.984778 11110 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:01:24.985152 11110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "bac5683e28774f5cbe0621383f97f900"
format_stamp: "Formatted at 2026-09-29 01:01:24 on dist-test-slave-k0b7"
server_key: "35f09b8ec26107cc5978041498582006"
server_key_iv: "96e54ad25d2edd7851bc62363d3409df"
server_key_version: "encryptionkey@0"
I20260929 01:01:24.985610 11110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:25.025221 11110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:25.030450 11110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:25.030821 11110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:25.034507 11110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:25.034643 11110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:01:25.034821 11110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:25.034955 11110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:01:25 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643684, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:25.071383 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.035110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:54265 (local address 127.3.21.254:38723)
0929 01:01:25.035630 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.035639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.037210 (+ 1571us) server_negotiation.cc:407] Connection header received
0929 01:01:25.040277 (+ 3067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.040284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.040398 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.040568 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:25.043321 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.044738 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.048469 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.049716 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.050081 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.057965 (+ 7884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:25.057977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:25.057985 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:25.058046 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:25.062638 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:25.064476 (+ 1838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:25.064486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:25.064493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:25.064732 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:25.065758 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:25.065768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:25.065774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:25.066262 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:25.066514 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.070826 (+ 4312us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.071086 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
I20260929 01:01:25.071504 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.036000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.3.21.195:54265)
0929 01:01:25.037003 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.037052 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.039739 (+ 2687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.040848 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.040858 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.041985 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:25.042999 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.043064 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.045126 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.045136 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.048194 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.048228 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.050543 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.050559 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.051243 (+ 684us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.052538 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:25.052590 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:25.057484 (+ 4894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:25.063045 (+ 5561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:25.063060 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:25.063092 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:25.064209 (+ 1117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:25.064946 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:25.064953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:25.064965 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:25.065487 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:25.066605 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:25.066621 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:25.067469 (+ 848us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.067965 (+ 496us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.068880 (+ 915us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":439,"threads_started":1}
I20260929 01:01:25.099131 11110 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37943
I20260929 01:01:25.099186 11245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37943 every 8 connection(s)
I20260929 01:01:25.101070 11110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:25.102136 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.081200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:44983 (local address 127.3.21.254:38723)
0929 01:01:25.081415 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.081423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.082917 (+ 1494us) server_negotiation.cc:407] Connection header received
0929 01:01:25.083207 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.083211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.083279 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.083377 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:25.085620 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.086391 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.089498 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.090184 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.090402 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.093979 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:25.093985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:25.093989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:25.094025 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:25.098635 (+ 4610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:25.099765 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:25.099771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:25.099783 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:25.099974 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:25.100396 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:25.100403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:25.100410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:25.100836 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:25.101016 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.101502 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.101755 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:01:25.102367 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.081256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.3.21.195:44983)
0929 01:01:25.082760 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.082798 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.083033 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.083469 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.083477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.084428 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:25.085270 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.085420 (+ 150us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.086646 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.086656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.089227 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.089255 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.090541 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.090552 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.090874 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.091949 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:25.091986 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:25.093564 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:25.098867 (+ 5303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:25.098876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:25.098883 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:25.099578 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:25.100110 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:25.100114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:25.100116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:25.100249 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:25.100974 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:25.100977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:25.101153 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.101306 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.101731 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"thread_start_us":883,"threads_started":1}
I20260929 01:01:25.102463 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11110
I20260929 01:01:25.102649 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:01:25.103307 3159 external_mini_cluster.cc:1428] Setting key 1fdab1a4e84b2de673522e3eb2720a2c
I20260929 01:01:25.105036 11246 heartbeater.cc:347] Connected to a master server at 127.3.21.254:38723
I20260929 01:01:25.105456 11246 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:25.106415 11246 heartbeater.cc:511] Master 127.3.21.254:38723 requested a full tablet report, sending...
I20260929 01:01:25.108105 10758 ts_manager.cc:194] Registered new tserver with Master: bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:25.109028 10758 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:54265
I20260929 01:01:25.120071 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:01:25 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:01:25 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:25 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:01:25.179682 11134 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:25.165686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.3.21.254:38723)
0929 01:01:25.165941 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.165948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.166121 (+ 173us) server_negotiation.cc:407] Connection header received
0929 01:01:25.166537 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.166551 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.166625 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.166746 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:25.168349 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.169235 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.171238 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.171781 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.171918 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.179248 (+ 7330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49588: BlockingRecv error: recv got EOF from 127.0.0.1:49588 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 01:01:25 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:25.200316 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.183717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.3.21.254:38723)
0929 01:01:25.183994 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.184001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.184281 (+ 280us) server_negotiation.cc:407] Connection header received
0929 01:01:25.184357 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.184361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.184429 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.184546 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:25.186014 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.186831 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.188652 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.189119 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.189235 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.193940 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:25.193954 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:25.193959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:25.194008 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:25.196908 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:25.197790 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:25.197794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:25.197797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:25.197934 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:25.198476 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:25.198550 (+ 74us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:25.198554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:25.198895 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:25.199130 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.199684 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.199888 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 01:01:25.205897 10758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:01:25.213989 10758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:25.247382 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.230847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:34755 (local address 127.0.0.1:56640)
0929 01:01:25.232447 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.232507 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.232774 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.235100 (+ 2326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.235113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.235156 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:25.235796 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.235815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.239391 (+ 3576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.239402 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.243611 (+ 4209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.243649 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.246295 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.246302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.246563 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.246639 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.246781 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.247006 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"thread_start_us":3234,"threads_started":1}
I20260929 01:01:25.247650 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.231680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.3.21.193:34755)
0929 01:01:25.234119 (+ 2439us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.234127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.234162 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:01:25.234298 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.234307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.234739 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.236103 (+ 1364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:25.236587 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.239111 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.244012 (+ 4901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.245956 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.246245 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.246604 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.247037 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.247331 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2231,"thread_start_us":2053,"threads_started":1}
I20260929 01:01:25.249068 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.231226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.3.21.194:37045)
0929 01:01:25.233523 (+ 2297us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.233538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.234537 (+ 999us) server_negotiation.cc:407] Connection header received
0929 01:01:25.234868 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.234876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.235108 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.235475 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:25.236280 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.240783 (+ 4503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.245960 (+ 5177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.247555 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.247696 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.247917 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.248547 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.248739 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2055,"thread_start_us":1869,"threads_started":1}
I20260929 01:01:25.250429 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.230998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:37045 (local address 127.0.0.1:42794)
0929 01:01:25.234394 (+ 3396us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.234415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.234647 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.235710 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.235713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.235737 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:25.236092 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.236100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.241041 (+ 4941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.241050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.245727 (+ 4677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.245754 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.248115 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.248122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.248338 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.248375 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.248493 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.250054 (+ 1561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3268,"mutex_wait_us":3,"thread_start_us":855,"threads_started":1}
I20260929 01:01:25.252907 10896 tablet_service.cc:1467] Processing CreateTablet for tablet d9b8f59bc80046a298a8e22492c6d455 (DEFAULT_TABLE table=test-table [id=7eff48272b604628b9e9324e0943d707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:25.253818 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.236555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37943 (local address 127.0.0.1:45044)
0929 01:01:25.244819 (+ 8264us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.244858 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.245030 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.245807 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.245812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.245845 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:25.246299 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.246313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.248714 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.248719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.251158 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.251179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.252836 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.252843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.253063 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.253111 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.253245 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.253424 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8118,"thread_start_us":7930,"threads_started":1}
I20260929 01:01:25.253835 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.234955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.3.21.195:37943)
0929 01:01:25.244462 (+ 9507us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.244472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.245006 (+ 534us) server_negotiation.cc:407] Connection header received
0929 01:01:25.245221 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.245231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.245536 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.245776 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:25.246620 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.248492 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.251393 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.252601 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.252781 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.253054 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.253303 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.253475 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9300,"thread_start_us":9088,"threads_started":1}
I20260929 01:01:25.254971 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b8f59bc80046a298a8e22492c6d455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:25.254673 11036 tablet_service.cc:1467] Processing CreateTablet for tablet d9b8f59bc80046a298a8e22492c6d455 (DEFAULT_TABLE table=test-table [id=7eff48272b604628b9e9324e0943d707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:25.256649 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b8f59bc80046a298a8e22492c6d455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:25.258179 11176 tablet_service.cc:1467] Processing CreateTablet for tablet d9b8f59bc80046a298a8e22492c6d455 (DEFAULT_TABLE table=test-table [id=7eff48272b604628b9e9324e0943d707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:25.259605 11176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b8f59bc80046a298a8e22492c6d455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:25.274868 11271 tablet_bootstrap.cc:492] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Bootstrap starting.
I20260929 01:01:25.275996 11270 tablet_bootstrap.cc:492] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Bootstrap starting.
I20260929 01:01:25.277439 11272 tablet_bootstrap.cc:492] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Bootstrap starting.
I20260929 01:01:25.280171 11270 tablet_bootstrap.cc:654] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:25.280714 11271 tablet_bootstrap.cc:654] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:25.281124 11272 tablet_bootstrap.cc:654] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:25.281607 11270 log.cc:824] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:25.282008 11271 log.cc:824] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:25.282274 11272 log.cc:824] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:25.286583 11272 tablet_bootstrap.cc:492] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: No bootstrap required, opened a new log
I20260929 01:01:25.286645 11270 tablet_bootstrap.cc:492] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: No bootstrap required, opened a new log
I20260929 01:01:25.286937 11270 ts_tablet_manager.cc:1397] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260929 01:01:25.286957 11272 ts_tablet_manager.cc:1397] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 01:01:25.287925 11271 tablet_bootstrap.cc:492] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: No bootstrap required, opened a new log
I20260929 01:01:25.288214 11271 ts_tablet_manager.cc:1397] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260929 01:01:25.295331 11272 raft_consensus.cc:348] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.295807 11272 raft_consensus.cc:374] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:25.295913 11272 raft_consensus.cc:729] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5683e28774f5cbe0621383f97f900, State: Initialized, Role: FOLLOWER
I20260929 01:01:25.296686 11272 consensus_queue.cc:260] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.298826 11270 raft_consensus.cc:348] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.299315 11246 heartbeater.cc:503] Master 127.3.21.254:38723 was elected leader, sending a full tablet report...
I20260929 01:01:25.299366 11270 raft_consensus.cc:374] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:25.299468 11270 raft_consensus.cc:729] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbadfda961147e4875ab1359c520999, State: Initialized, Role: FOLLOWER
I20260929 01:01:25.299715 11271 raft_consensus.cc:348] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.300093 11272 ts_tablet_manager.cc:1428] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260929 01:01:25.300140 11271 raft_consensus.cc:374] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:25.300221 11271 raft_consensus.cc:729] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Initialized, Role: FOLLOWER
I20260929 01:01:25.300125 11270 consensus_queue.cc:260] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.301002 11271 consensus_queue.cc:260] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.302143 11270 ts_tablet_manager.cc:1428] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 01:01:25.302245 11106 heartbeater.cc:503] Master 127.3.21.254:38723 was elected leader, sending a full tablet report...
I20260929 01:01:25.304342 11271 ts_tablet_manager.cc:1428] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260929 01:01:25.354242 11247 tablet.cc:2369] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:01:25.365607 10967 tablet.cc:2369] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:01:25.486810 11107 tablet.cc:2369] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:01:25.558856 11278 raft_consensus.cc:482] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:25.559257 11278 raft_consensus.cc:504] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.561100 11278 leader_election.cc:288] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:25.570936 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.562020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:52497 (local address 127.3.21.193:34755)
0929 01:01:25.562281 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.562288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.562572 (+ 284us) server_negotiation.cc:407] Connection header received
0929 01:01:25.562864 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.562871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.562994 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.563127 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:25.563872 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.564857 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.567757 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.569374 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.569587 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.569684 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.570456 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.570683 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 01:01:25.570991 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.562889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:55031 (local address 127.3.21.195:37943)
0929 01:01:25.563083 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.563090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.563616 (+ 526us) server_negotiation.cc:407] Connection header received
0929 01:01:25.563844 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.563850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.563927 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.564039 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:25.564718 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.565523 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.567981 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.569374 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.569587 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.569684 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.570635 (+ 951us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.570772 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:01:25.571162 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.562673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37943 (local address 127.3.21.194:55031)
0929 01:01:25.563463 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.563482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.563628 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.564091 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.564101 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.564122 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:25.564544 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.564555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.565661 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.565667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.567787 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.567797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.569799 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.569809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.570111 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.570403 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.570548 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.570679 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":166,"thread_start_us":405,"threads_started":1}
I20260929 01:01:25.571205 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.561598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:34755 (local address 127.3.21.194:52497)
0929 01:01:25.562422 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:25.562439 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:25.562657 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:25.563189 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:25.563197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:25.563226 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:25.563661 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.563675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.565033 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.565037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.567508 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:25.567527 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.569861 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.569869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:25.570109 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.570264 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 01:01:25.570395 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 01:01:25.570620 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":94,"thread_start_us":310,"threads_started":1}
I20260929 01:01:25.572438 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b8f59bc80046a298a8e22492c6d455" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900" is_pre_election: true
I20260929 01:01:25.572734 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b8f59bc80046a298a8e22492c6d455" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999" is_pre_election: true
I20260929 01:01:25.573017 11196 raft_consensus.cc:2454] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:25.573331 10916 raft_consensus.cc:2454] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:25.573933 10987 leader_election.cc:302] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, bac5683e28774f5cbe0621383f97f900; no voters:
I20260929 01:01:25.574442 11278 raft_consensus.cc:2781] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:25.574709 11278 raft_consensus.cc:482] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:25.574854 11278 raft_consensus.cc:3037] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:25.577555 11278 raft_consensus.cc:504] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.578277 11278 leader_election.cc:288] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Requested vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:25.579069 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b8f59bc80046a298a8e22492c6d455" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999"
I20260929 01:01:25.579263 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b8f59bc80046a298a8e22492c6d455" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900"
I20260929 01:01:25.579479 10916 raft_consensus.cc:3037] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:25.579691 11196 raft_consensus.cc:3037] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:25.582267 10916 raft_consensus.cc:2454] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:25.582408 11196 raft_consensus.cc:2454] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:25.582878 10987 leader_election.cc:302] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, cbbadfda961147e4875ab1359c520999; no voters:
I20260929 01:01:25.583323 11278 raft_consensus.cc:2781] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:25.583899 11278 raft_consensus.cc:686] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Becoming Leader. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Running, Role: LEADER
I20260929 01:01:25.584451 11278 consensus_queue.cc:237] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.589907 10758 catalog_manager.cc:5495] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 874188f28c5449a786ccb82684ed4e56 (127.3.21.194). New cstate: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260929 01:01:25.616698 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:25.610637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.3.21.194:37045)
0929 01:01:25.610824 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:25.610828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:25.610847 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:25.611031 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:25.611035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:25.611121 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:25.611265 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:25.611856 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.612670 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.614139 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:25.614613 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:25.614714 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:25.615285 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:25.615930 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:25.616059 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:25.616237 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 01:01:25.616319 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:01:25.626755 11196 raft_consensus.cc:1261] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:25.626847 10916 raft_consensus.cc:1261] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:25.627807 11281 consensus_queue.cc:1035] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:25.628401 11278 consensus_queue.cc:1035] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:25.660109 10758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:25.660400 10758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:25.664832 10758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Sending DeleteTablet for 3 replicas of tablet d9b8f59bc80046a298a8e22492c6d455
I20260929 01:01:25.666538 11036 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b8f59bc80046a298a8e22492c6d455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:25 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:42794
I20260929 01:01:25.666448 11176 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b8f59bc80046a298a8e22492c6d455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:25 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:45044
I20260929 01:01:25.667089 10896 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b8f59bc80046a298a8e22492c6d455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:25 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260929 01:01:25.668666 10758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:01:25.670403 10758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:25.675117 11296 tablet_replica.cc:330] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: stopping tablet replica
I20260929 01:01:25.675859 11294 tablet_replica.cc:330] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: stopping tablet replica
I20260929 01:01:25.676491 11294 raft_consensus.cc:2229] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:25.677093 11296 raft_consensus.cc:2229] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:25.677771 11294 raft_consensus.cc:2258] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:25.678337 11296 raft_consensus.cc:2258] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:25.680896 11296 ts_tablet_manager.cc:1905] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:25.683699 11035 tablet_service.cc:1467] Processing CreateTablet for tablet 920b02994652413fb925c6af6ca7dce9 (DEFAULT_TABLE table=test-table [id=43476da206f34efe803b5b292aaef9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:25.684360 11035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920b02994652413fb925c6af6ca7dce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:25.690591 11175 tablet_service.cc:1467] Processing CreateTablet for tablet 920b02994652413fb925c6af6ca7dce9 (DEFAULT_TABLE table=test-table [id=43476da206f34efe803b5b292aaef9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:25.683895 10896 tablet_service.cc:1467] Processing CreateTablet for tablet 920b02994652413fb925c6af6ca7dce9 (DEFAULT_TABLE table=test-table [id=43476da206f34efe803b5b292aaef9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:25.691285 11175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920b02994652413fb925c6af6ca7dce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:25.691707 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920b02994652413fb925c6af6ca7dce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:25.699513 11296 ts_tablet_manager.cc:1918] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:25.699839 11296 log.cc:1196] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/d9b8f59bc80046a298a8e22492c6d455
I20260929 01:01:25.700578 11272 tablet_bootstrap.cc:492] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Bootstrap starting.
I20260929 01:01:25.691278 11294 ts_tablet_manager.cc:1905] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:25.701057 11270 tablet_bootstrap.cc:492] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Bootstrap starting.
I20260929 01:01:25.702504 11272 tablet_bootstrap.cc:654] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:25.702840 11270 tablet_bootstrap.cc:654] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:25.700594 11296 ts_tablet_manager.cc:1939] T d9b8f59bc80046a298a8e22492c6d455 P cbbadfda961147e4875ab1359c520999: Deleting consensus metadata
I20260929 01:01:25.705159 10744 catalog_manager.cc:4841] TS cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755): tablet d9b8f59bc80046a298a8e22492c6d455 (table test-table [id=7eff48272b604628b9e9324e0943d707]) successfully deleted
I20260929 01:01:25.706034 11271 tablet_bootstrap.cc:492] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Bootstrap starting.
I20260929 01:01:25.708557 11271 tablet_bootstrap.cc:654] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:25.710441 11294 ts_tablet_manager.cc:1918] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:25.710675 11294 log.cc:1196] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/d9b8f59bc80046a298a8e22492c6d455
I20260929 01:01:25.711295 11294 ts_tablet_manager.cc:1939] T d9b8f59bc80046a298a8e22492c6d455 P bac5683e28774f5cbe0621383f97f900: Deleting consensus metadata
I20260929 01:01:25.711998 11295 tablet_replica.cc:330] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: stopping tablet replica
I20260929 01:01:25.713328 11295 raft_consensus.cc:2229] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:01:25.713663 10744 catalog_manager.cc:4841] TS bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943): tablet d9b8f59bc80046a298a8e22492c6d455 (table test-table [id=7eff48272b604628b9e9324e0943d707]) successfully deleted
I20260929 01:01:25.715524 11272 tablet_bootstrap.cc:492] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: No bootstrap required, opened a new log
I20260929 01:01:25.715688 11272 ts_tablet_manager.cc:1397] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.001s
I20260929 01:01:25.716434 11295 raft_consensus.cc:2258] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:25.716528 11272 raft_consensus.cc:348] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.715790 11271 tablet_bootstrap.cc:492] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: No bootstrap required, opened a new log
I20260929 01:01:25.716744 11272 raft_consensus.cc:374] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:25.716840 11272 raft_consensus.cc:729] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5683e28774f5cbe0621383f97f900, State: Initialized, Role: FOLLOWER
I20260929 01:01:25.716908 11271 ts_tablet_manager.cc:1397] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260929 01:01:25.717159 11272 consensus_queue.cc:260] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.717500 11270 tablet_bootstrap.cc:492] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: No bootstrap required, opened a new log
I20260929 01:01:25.717676 11270 ts_tablet_manager.cc:1397] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20260929 01:01:25.717702 11271 raft_consensus.cc:348] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.717922 11271 raft_consensus.cc:374] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:25.718027 11271 raft_consensus.cc:729] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Initialized, Role: FOLLOWER
I20260929 01:01:25.718297 11271 consensus_queue.cc:260] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.718917 11271 ts_tablet_manager.cc:1428] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:25.719377 11272 ts_tablet_manager.cc:1428] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 01:01:25.719446 11270 raft_consensus.cc:348] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.719705 11270 raft_consensus.cc:374] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:25.719821 11270 raft_consensus.cc:729] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbadfda961147e4875ab1359c520999, State: Initialized, Role: FOLLOWER
I20260929 01:01:25.720196 11270 consensus_queue.cc:260] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.721194 11295 ts_tablet_manager.cc:1905] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:25.721724 11270 ts_tablet_manager.cc:1428] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:25.728952 11295 ts_tablet_manager.cc:1918] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:25.729183 11295 log.cc:1196] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/d9b8f59bc80046a298a8e22492c6d455
I20260929 01:01:25.729830 11295 ts_tablet_manager.cc:1939] T d9b8f59bc80046a298a8e22492c6d455 P 874188f28c5449a786ccb82684ed4e56: Deleting consensus metadata
I20260929 01:01:25.731403 10746 catalog_manager.cc:4841] TS 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045): tablet d9b8f59bc80046a298a8e22492c6d455 (table test-table [id=7eff48272b604628b9e9324e0943d707]) successfully deleted
I20260929 01:01:25.940141 11281 raft_consensus.cc:482] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:25.940455 11281 raft_consensus.cc:504] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.941345 11281 leader_election.cc:288] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:25.941994 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b02994652413fb925c6af6ca7dce9" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999" is_pre_election: true
I20260929 01:01:25.942081 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b02994652413fb925c6af6ca7dce9" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900" is_pre_election: true
I20260929 01:01:25.942277 10916 raft_consensus.cc:2454] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:25.942298 11196 raft_consensus.cc:2454] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:25.942775 10987 leader_election.cc:302] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, bac5683e28774f5cbe0621383f97f900; no voters:
I20260929 01:01:25.943097 11281 raft_consensus.cc:2781] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:25.943228 11281 raft_consensus.cc:482] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:25.943359 11281 raft_consensus.cc:3037] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:25.947252 11281 raft_consensus.cc:504] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.948334 11281 leader_election.cc:288] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Requested vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:25.948940 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b02994652413fb925c6af6ca7dce9" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999"
I20260929 01:01:25.948987 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920b02994652413fb925c6af6ca7dce9" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900"
I20260929 01:01:25.949172 10916 raft_consensus.cc:3037] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:25.949213 11196 raft_consensus.cc:3037] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:25.951905 10916 raft_consensus.cc:2454] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:25.952309 11196 raft_consensus.cc:2454] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:25.952590 10987 leader_election.cc:302] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, cbbadfda961147e4875ab1359c520999; no voters:
I20260929 01:01:25.953007 11281 raft_consensus.cc:2781] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:25.953298 11281 raft_consensus.cc:686] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Becoming Leader. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Running, Role: LEADER
I20260929 01:01:25.953588 11281 consensus_queue.cc:237] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:25.957144 10757 catalog_manager.cc:5495] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 874188f28c5449a786ccb82684ed4e56 (127.3.21.194). New cstate: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:01:26 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643683, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.254@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:26.051443 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.033246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.3.21.254:38723)
0929 01:01:26.034567 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.034572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.034599 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:01:26.034639 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.034642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.034701 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.034816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:26.036146 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.036952 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.038210 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.038734 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.039118 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.044458 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:26.044466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:26.044470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:26.044515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:26.048263 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:26.049277 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:26.049285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:26.049288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:26.049419 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:26.049806 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:26.049813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:26.049817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:26.050099 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:26.050256 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.050725 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.051008 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":1157,"thread_start_us":929,"threads_started":1}
I20260929 01:01:26.051524 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.033137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:38723 (local address 127.0.0.1:49602)
0929 01:01:26.034184 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.034214 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.034413 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.034870 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.034875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.035738 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:26.035968 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.035978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.037103 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.037111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.038061 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.038071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.039136 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.039142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.039547 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.040578 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:26.040596 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:26.043762 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:26.048573 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:26.048605 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:26.048611 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:26.049149 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:26.049541 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:26.049553 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:26.049555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:26.049698 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:26.050246 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:26.050255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:26.050416 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.050650 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.051127 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":238,"thread_start_us":710,"threads_started":1}
I20260929 01:01:26.057807 10757 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49602:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:01:26.059017 10757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:26.066980 11036 tablet_service.cc:1467] Processing CreateTablet for tablet 42e16e2caef04da5b1cacd856303bb8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ab46c2c6fd14d77bc6bd2024b8b1434]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:01:26.067080 10896 tablet_service.cc:1467] Processing CreateTablet for tablet 42e16e2caef04da5b1cacd856303bb8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ab46c2c6fd14d77bc6bd2024b8b1434]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:01:26.067596 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e16e2caef04da5b1cacd856303bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.067668 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e16e2caef04da5b1cacd856303bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.071617 11175 tablet_service.cc:1467] Processing CreateTablet for tablet 42e16e2caef04da5b1cacd856303bb8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ab46c2c6fd14d77bc6bd2024b8b1434]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:01:26.072186 11175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e16e2caef04da5b1cacd856303bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.075018 11271 tablet_bootstrap.cc:492] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: Bootstrap starting.
I20260929 01:01:26.075816 11270 tablet_bootstrap.cc:492] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: Bootstrap starting.
I20260929 01:01:26.076788 11271 tablet_bootstrap.cc:654] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.077714 11270 tablet_bootstrap.cc:654] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.082226 11272 tablet_bootstrap.cc:492] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: Bootstrap starting.
I20260929 01:01:26.084856 11272 tablet_bootstrap.cc:654] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.090128 11270 tablet_bootstrap.cc:492] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: No bootstrap required, opened a new log
I20260929 01:01:26.090358 11270 ts_tablet_manager.cc:1397] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.001s
I20260929 01:01:26.091467 11270 raft_consensus.cc:348] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.091771 11270 raft_consensus.cc:374] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.091874 11270 raft_consensus.cc:729] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbadfda961147e4875ab1359c520999, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.092212 11270 consensus_queue.cc:260] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.092825 11277 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.093237 11277 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.093777 11270 ts_tablet_manager.cc:1428] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 01:01:26.101739 11272 tablet_bootstrap.cc:492] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: No bootstrap required, opened a new log
I20260929 01:01:26.101971 11272 ts_tablet_manager.cc:1397] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20260929 01:01:26.102847 11272 raft_consensus.cc:348] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.103084 11272 raft_consensus.cc:374] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.103169 11272 raft_consensus.cc:729] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5683e28774f5cbe0621383f97f900, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.103448 11272 consensus_queue.cc:260] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.104099 11272 ts_tablet_manager.cc:1428] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 01:01:26.103974 11276 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.104372 11276 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.105398 11271 tablet_bootstrap.cc:492] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: No bootstrap required, opened a new log
I20260929 01:01:26.105626 11271 ts_tablet_manager.cc:1397] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.006s
I20260929 01:01:26.106468 11271 raft_consensus.cc:348] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.106799 11271 raft_consensus.cc:374] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.106900 11271 raft_consensus.cc:729] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.107218 11271 consensus_queue.cc:260] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.107962 11271 ts_tablet_manager.cc:1428] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 01:01:26.107818 11281 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.108174 11281 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.136103 11281 raft_consensus.cc:482] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:26.136310 11281 raft_consensus.cc:504] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.137039 11281 leader_election.cc:288] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:26.138327 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e16e2caef04da5b1cacd856303bb8f" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999" is_pre_election: true
I20260929 01:01:26.138684 10916 raft_consensus.cc:2454] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:26.139019 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e16e2caef04da5b1cacd856303bb8f" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900" is_pre_election: true
I20260929 01:01:26.139281 11196 raft_consensus.cc:2454] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:26.139292 10987 leader_election.cc:302] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, cbbadfda961147e4875ab1359c520999; no voters:
I20260929 01:01:26.139706 11281 raft_consensus.cc:2781] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:26.139854 11281 raft_consensus.cc:482] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:26.139971 11281 raft_consensus.cc:3037] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.144515 11281 raft_consensus.cc:504] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.146780 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e16e2caef04da5b1cacd856303bb8f" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999"
I20260929 01:01:26.147081 10916 raft_consensus.cc:3037] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.149335 11281 leader_election.cc:288] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Requested vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:26.150228 10916 raft_consensus.cc:2454] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:26.150362 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e16e2caef04da5b1cacd856303bb8f" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900"
I20260929 01:01:26.150660 11196 raft_consensus.cc:3037] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.153735 11196 raft_consensus.cc:2454] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:26.154691 10987 leader_election.cc:302] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, bac5683e28774f5cbe0621383f97f900; no voters:
I20260929 01:01:26.155149 11281 raft_consensus.cc:2781] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:26.155365 11281 raft_consensus.cc:686] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Becoming Leader. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Running, Role: LEADER
I20260929 01:01:26.155740 11281 consensus_queue.cc:237] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.157631 11278 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: TxnStatusTablet state changed. Reason: New leader 874188f28c5449a786ccb82684ed4e56. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.157898 11278 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:01:26.158969 11312 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:01:26.159235 11312 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:01:26.159588 10759 catalog_manager.cc:5495] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 874188f28c5449a786ccb82684ed4e56 (127.3.21.194). New cstate: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260929 01:01:26.184489 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.174462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.3.21.194:37045)
0929 01:01:26.178988 (+ 4526us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.178993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.179017 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:01:26.179083 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.179088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.179158 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.179302 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:26.179771 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.180807 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.182420 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.182971 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.183116 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.183614 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.183746 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.183852 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.184021 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.184102 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4399,"thread_start_us":4153,"threads_started":1}
I20260929 01:01:26.184473 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.173966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:37045 (local address 127.0.0.1:42822)
0929 01:01:26.174263 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.174288 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.174458 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.179357 (+ 4899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.179362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.179376 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:01:26.179589 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.179599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.180953 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.180960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.182250 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.182265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.183137 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.183144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.183331 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.183413 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.183862 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.183887 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.183991 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.184147 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 01:01:26.189059 10916 raft_consensus.cc:1261] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.189069 11196 raft_consensus.cc:1261] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.189711 11278 consensus_queue.cc:1035] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.189980 11281 consensus_queue.cc:1035] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.208635 11276 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: TxnStatusTablet state changed. Reason: New leader 874188f28c5449a786ccb82684ed4e56. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.208916 11276 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.211483 11277 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: TxnStatusTablet state changed. Reason: New leader 874188f28c5449a786ccb82684ed4e56. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.211799 11277 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.217521 11278 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.217869 11278 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:01:26.218990 11276 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.219267 11276 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P bac5683e28774f5cbe0621383f97f900: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.221194 11278 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.221470 11278 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P 874188f28c5449a786ccb82684ed4e56: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:01:26.226657 11277 tablet_replica.cc:439] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } } }
I20260929 01:01:26.226953 11277 tablet_replica.cc:442] T 42e16e2caef04da5b1cacd856303bb8f P cbbadfda961147e4875ab1359c520999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:01:26.246397 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.238603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:52211 (local address 127.3.21.194:37045)
0929 01:01:26.238801 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.238807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.240129 (+ 1322us) server_negotiation.cc:407] Connection header received
0929 01:01:26.240444 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.240449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.240531 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.240637 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:26.241344 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.242523 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.244100 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.244785 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.244934 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.245449 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.245584 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.245686 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.245886 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.245969 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
I20260929 01:01:26.246572 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.238661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:37045 (local address 127.3.21.194:52211)
0929 01:01:26.239995 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.240016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.240175 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.240661 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.240665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.240677 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:01:26.241161 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.241171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.242659 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.242664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.243950 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.243962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.245031 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.245036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.245174 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.245252 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.245730 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.245742 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.245848 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.245976 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":51,"thread_start_us":983,"threads_started":1}
I20260929 01:01:26.257768 10916 raft_consensus.cc:1261] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.257884 11196 raft_consensus.cc:1261] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.258644 11326 consensus_queue.cc:1035] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.259090 11281 consensus_queue.cc:1035] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.266678 11315 mvcc.cc:205] Tried to move back new op lower bound from 7334476538905427968 to 7334476537668878336. Current Snapshot: MvccSnapshot[applied={T|T < 7334476538905427968}]
I20260929 01:01:26.270100 11317 mvcc.cc:205] Tried to move back new op lower bound from 7334476538905427968 to 7334476537668878336. Current Snapshot: MvccSnapshot[applied={T|T < 7334476538905427968}]
I20260929 01:01:26.343150 10759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:26.343492 10759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:26.347780 10759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Sending DeleteTablet for 3 replicas of tablet 920b02994652413fb925c6af6ca7dce9
I20260929 01:01:26.348878 11035 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b02994652413fb925c6af6ca7dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:42794
I20260929 01:01:26.348904 10896 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b02994652413fb925c6af6ca7dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260929 01:01:26.349021 11175 tablet_service.cc:1514] Processing DeleteTablet for tablet 920b02994652413fb925c6af6ca7dce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:45044
I20260929 01:01:26.364348 11330 tablet_replica.cc:330] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: stopping tablet replica
I20260929 01:01:26.364796 11330 raft_consensus.cc:2229] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:26.365170 11330 raft_consensus.cc:2258] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.366719 11330 ts_tablet_manager.cc:1905] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.376734 11330 ts_tablet_manager.cc:1918] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:01:26.377048 11330 log.cc:1196] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/920b02994652413fb925c6af6ca7dce9
I20260929 01:01:26.377688 11330 ts_tablet_manager.cc:1939] T 920b02994652413fb925c6af6ca7dce9 P cbbadfda961147e4875ab1359c520999: Deleting consensus metadata
I20260929 01:01:26.377899 11331 tablet_replica.cc:330] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: stopping tablet replica
I20260929 01:01:26.378055 11332 tablet_replica.cc:330] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: stopping tablet replica
I20260929 01:01:26.378146 11331 raft_consensus.cc:2229] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:01:26.378544 11332 raft_consensus.cc:2229] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:26.378635 11331 raft_consensus.cc:2258] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.378829 11332 raft_consensus.cc:2258] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.379557 10744 catalog_manager.cc:4841] TS cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755): tablet 920b02994652413fb925c6af6ca7dce9 (table test-table [id=43476da206f34efe803b5b292aaef9d4]) successfully deleted
I20260929 01:01:26.380119 11332 ts_tablet_manager.cc:1905] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.381016 11331 ts_tablet_manager.cc:1905] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.388907 11332 ts_tablet_manager.cc:1918] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:01:26.389068 11332 log.cc:1196] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/920b02994652413fb925c6af6ca7dce9
I20260929 01:01:26.389526 11332 ts_tablet_manager.cc:1939] T 920b02994652413fb925c6af6ca7dce9 P bac5683e28774f5cbe0621383f97f900: Deleting consensus metadata
I20260929 01:01:26.390003 11331 ts_tablet_manager.cc:1918] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:01:26.390232 11331 log.cc:1196] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/920b02994652413fb925c6af6ca7dce9
I20260929 01:01:26.390785 10744 catalog_manager.cc:4841] TS bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943): tablet 920b02994652413fb925c6af6ca7dce9 (table test-table [id=43476da206f34efe803b5b292aaef9d4]) successfully deleted
I20260929 01:01:26.391388 11331 ts_tablet_manager.cc:1939] T 920b02994652413fb925c6af6ca7dce9 P 874188f28c5449a786ccb82684ed4e56: Deleting consensus metadata
I20260929 01:01:26.392562 10746 catalog_manager.cc:4841] TS 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045): tablet 920b02994652413fb925c6af6ca7dce9 (table test-table [id=43476da206f34efe803b5b292aaef9d4]) successfully deleted
Sep 29 01:01:26 dist-test-slave-k0b7 krb5kdc[10714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:01:26 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:26 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:01:26.434473 11307 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:26.422264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.3.21.254:38723)
0929 01:01:26.422609 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.422619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.423124 (+ 505us) server_negotiation.cc:407] Connection header received
0929 01:01:26.423295 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.423300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.423376 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.423521 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:26.425100 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.425857 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.427193 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.427619 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.427770 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.434164 (+ 6394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49606: BlockingRecv error: recv got EOF from 127.0.0.1:49606 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 01:01:26 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM
I20260929 01:01:26.457617 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.437738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.3.21.254:38723)
0929 01:01:26.438007 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.438015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.438579 (+ 564us) server_negotiation.cc:407] Connection header received
0929 01:01:26.438737 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.438741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.438866 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.439118 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:26.441152 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.442184 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.444751 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.445481 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.445649 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.451396 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:26.451402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:26.451406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:26.451454 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:26.454469 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:26.455388 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:26.455394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:26.455405 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:26.455562 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:26.456004 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:26.456017 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:26.456019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:26.456322 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:26.456625 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.456968 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.457237 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:01:26.461784 10759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:01:26.462388 10759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:26.469583 10896 tablet_service.cc:1467] Processing CreateTablet for tablet 5673ec793e634ab5aa65174162eec4dd (DEFAULT_TABLE table=test-table [id=6020622331d94f9cb9a4af2b2daaa89f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:26.469969 11175 tablet_service.cc:1467] Processing CreateTablet for tablet 5673ec793e634ab5aa65174162eec4dd (DEFAULT_TABLE table=test-table [id=6020622331d94f9cb9a4af2b2daaa89f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:26.470090 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5673ec793e634ab5aa65174162eec4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.470414 11175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5673ec793e634ab5aa65174162eec4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.474174 11035 tablet_service.cc:1467] Processing CreateTablet for tablet 5673ec793e634ab5aa65174162eec4dd (DEFAULT_TABLE table=test-table [id=6020622331d94f9cb9a4af2b2daaa89f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:26.474669 11035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5673ec793e634ab5aa65174162eec4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.476966 11270 tablet_bootstrap.cc:492] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Bootstrap starting.
I20260929 01:01:26.478217 11272 tablet_bootstrap.cc:492] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Bootstrap starting.
I20260929 01:01:26.478631 11270 tablet_bootstrap.cc:654] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.480743 11272 tablet_bootstrap.cc:654] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.483968 11271 tablet_bootstrap.cc:492] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Bootstrap starting.
I20260929 01:01:26.485854 11271 tablet_bootstrap.cc:654] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.500875 11270 tablet_bootstrap.cc:492] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: No bootstrap required, opened a new log
I20260929 01:01:26.501049 11270 ts_tablet_manager.cc:1397] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20260929 01:01:26.501746 11271 tablet_bootstrap.cc:492] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: No bootstrap required, opened a new log
I20260929 01:01:26.501915 11271 ts_tablet_manager.cc:1397] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260929 01:01:26.501822 11270 raft_consensus.cc:348] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.502063 11270 raft_consensus.cc:374] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.502140 11270 raft_consensus.cc:729] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbadfda961147e4875ab1359c520999, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.502349 11270 consensus_queue.cc:260] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.502452 11272 tablet_bootstrap.cc:492] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: No bootstrap required, opened a new log
I20260929 01:01:26.502640 11272 ts_tablet_manager.cc:1397] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.006s
I20260929 01:01:26.502861 11270 ts_tablet_manager.cc:1428] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:01:26.502976 11271 raft_consensus.cc:348] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.503258 11271 raft_consensus.cc:374] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.503368 11271 raft_consensus.cc:729] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.503639 11271 consensus_queue.cc:260] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.503886 11272 raft_consensus.cc:348] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.504258 11272 raft_consensus.cc:374] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.504365 11272 raft_consensus.cc:729] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5683e28774f5cbe0621383f97f900, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.504595 11272 consensus_queue.cc:260] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.505235 11272 ts_tablet_manager.cc:1428] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 01:01:26.506531 11271 ts_tablet_manager.cc:1428] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 01:01:26.523975 11326 raft_consensus.cc:482] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:26.524314 11326 raft_consensus.cc:504] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.525053 11326 leader_election.cc:288] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:26.525835 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5673ec793e634ab5aa65174162eec4dd" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999" is_pre_election: true
I20260929 01:01:26.526080 10916 raft_consensus.cc:2454] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:26.526050 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5673ec793e634ab5aa65174162eec4dd" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900" is_pre_election: true
I20260929 01:01:26.526269 11196 raft_consensus.cc:2454] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 0.
I20260929 01:01:26.526821 10987 leader_election.cc:302] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, cbbadfda961147e4875ab1359c520999; no voters:
I20260929 01:01:26.527168 11326 raft_consensus.cc:2781] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:26.527275 11326 raft_consensus.cc:482] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:26.527390 11326 raft_consensus.cc:3037] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.530259 11326 raft_consensus.cc:504] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.531068 11326 leader_election.cc:288] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Requested vote from peers cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:26.531697 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5673ec793e634ab5aa65174162eec4dd" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900"
I20260929 01:01:26.531949 11196 raft_consensus.cc:3037] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.531906 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5673ec793e634ab5aa65174162eec4dd" candidate_uuid: "874188f28c5449a786ccb82684ed4e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbbadfda961147e4875ab1359c520999"
I20260929 01:01:26.532111 10916 raft_consensus.cc:3037] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.535091 10916 raft_consensus.cc:2454] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:26.535212 11196 raft_consensus.cc:2454] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 874188f28c5449a786ccb82684ed4e56 in term 1.
I20260929 01:01:26.535866 10987 leader_election.cc:302] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, bac5683e28774f5cbe0621383f97f900; no voters:
I20260929 01:01:26.536185 11326 raft_consensus.cc:2781] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:26.536362 11326 raft_consensus.cc:686] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Becoming Leader. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Running, Role: LEADER
I20260929 01:01:26.536685 11326 consensus_queue.cc:237] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.539836 10759 catalog_manager.cc:5495] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 874188f28c5449a786ccb82684ed4e56 (127.3.21.194). New cstate: current_term: 1 leader_uuid: "874188f28c5449a786ccb82684ed4e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260929 01:01:26.572146 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.565374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.3.21.194:37045)
0929 01:01:26.565633 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.565637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.565662 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:01:26.565951 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.565955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.566027 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.566134 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:26.566597 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.567515 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.569599 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.570151 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.570393 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.571072 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.571202 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.571286 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.571674 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.571808 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 01:01:26.575234 11196 raft_consensus.cc:1261] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.575230 10916 raft_consensus.cc:1261] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Refusing update from remote peer 874188f28c5449a786ccb82684ed4e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.575930 11326 consensus_queue.cc:1035] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.576260 11281 consensus_queue.cc:1035] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.582749 11316 mvcc.cc:205] Tried to move back new op lower bound from 7334476540205436928 to 7334476540057608192. Current Snapshot: MvccSnapshot[applied={T|T < 7334476540205436928}]
I20260929 01:01:26.589622 10757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:26.590005 10757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:26.594743 10757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Sending DeleteTablet for 3 replicas of tablet 5673ec793e634ab5aa65174162eec4dd
I20260929 01:01:26.595934 10896 tablet_service.cc:1514] Processing DeleteTablet for tablet 5673ec793e634ab5aa65174162eec4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260929 01:01:26.596263 11330 tablet_replica.cc:330] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: stopping tablet replica
I20260929 01:01:26.596485 11330 raft_consensus.cc:2229] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:26.597056 11330 raft_consensus.cc:2258] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.598109 11035 tablet_service.cc:1514] Processing DeleteTablet for tablet 5673ec793e634ab5aa65174162eec4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:42794
I20260929 01:01:26.598413 11331 tablet_replica.cc:330] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: stopping tablet replica
I20260929 01:01:26.598644 11331 raft_consensus.cc:2229] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:01:26.599007 11331 raft_consensus.cc:2258] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.599371 11175 tablet_service.cc:1514] Processing DeleteTablet for tablet 5673ec793e634ab5aa65174162eec4dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:45044
I20260929 01:01:26.599668 11332 tablet_replica.cc:330] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: stopping tablet replica
I20260929 01:01:26.599887 11330 ts_tablet_manager.cc:1905] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.599884 11332 raft_consensus.cc:2229] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:26.600103 11332 raft_consensus.cc:2258] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.600200 11331 ts_tablet_manager.cc:1905] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.606261 11331 ts_tablet_manager.cc:1918] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:26.606472 11331 log.cc:1196] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/5673ec793e634ab5aa65174162eec4dd
I20260929 01:01:26.606772 11330 ts_tablet_manager.cc:1918] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:26.606942 11330 log.cc:1196] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/5673ec793e634ab5aa65174162eec4dd
I20260929 01:01:26.606961 11331 ts_tablet_manager.cc:1939] T 5673ec793e634ab5aa65174162eec4dd P 874188f28c5449a786ccb82684ed4e56: Deleting consensus metadata
I20260929 01:01:26.607769 11330 ts_tablet_manager.cc:1939] T 5673ec793e634ab5aa65174162eec4dd P cbbadfda961147e4875ab1359c520999: Deleting consensus metadata
I20260929 01:01:26.608160 10746 catalog_manager.cc:4841] TS 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045): tablet 5673ec793e634ab5aa65174162eec4dd (table test-table [id=6020622331d94f9cb9a4af2b2daaa89f]) successfully deleted
I20260929 01:01:26.608420 10757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 01:01:26.608839 10744 catalog_manager.cc:4841] TS cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755): tablet 5673ec793e634ab5aa65174162eec4dd (table test-table [id=6020622331d94f9cb9a4af2b2daaa89f]) successfully deleted
W20260929 01:01:26.609126 10757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:01:26.610862 11332 ts_tablet_manager.cc:1905] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.617543 10896 tablet_service.cc:1467] Processing CreateTablet for tablet 7a7c1abca10d431dbceafd45bd842166 (DEFAULT_TABLE table=test-table [id=0254fe7b683d4ce8a5f8beb098569967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:26.617588 11175 tablet_service.cc:1467] Processing CreateTablet for tablet 7a7c1abca10d431dbceafd45bd842166 (DEFAULT_TABLE table=test-table [id=0254fe7b683d4ce8a5f8beb098569967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:26.617594 11035 tablet_service.cc:1467] Processing CreateTablet for tablet 7a7c1abca10d431dbceafd45bd842166 (DEFAULT_TABLE table=test-table [id=0254fe7b683d4ce8a5f8beb098569967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:01:26.618031 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a7c1abca10d431dbceafd45bd842166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.618049 11035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a7c1abca10d431dbceafd45bd842166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.618314 11175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a7c1abca10d431dbceafd45bd842166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:26.625869 11272 tablet_bootstrap.cc:492] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Bootstrap starting.
I20260929 01:01:26.625857 11271 tablet_bootstrap.cc:492] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Bootstrap starting.
I20260929 01:01:26.626368 11270 tablet_bootstrap.cc:492] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Bootstrap starting.
I20260929 01:01:26.627748 11272 tablet_bootstrap.cc:654] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.627736 11271 tablet_bootstrap.cc:654] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.628000 11270 tablet_bootstrap.cc:654] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:26.625581 11332 ts_tablet_manager.cc:1918] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:01:26.629753 11332 log.cc:1196] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/5673ec793e634ab5aa65174162eec4dd
I20260929 01:01:26.631074 11332 ts_tablet_manager.cc:1939] T 5673ec793e634ab5aa65174162eec4dd P bac5683e28774f5cbe0621383f97f900: Deleting consensus metadata
I20260929 01:01:26.632669 10744 catalog_manager.cc:4841] TS bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943): tablet 5673ec793e634ab5aa65174162eec4dd (table test-table [id=6020622331d94f9cb9a4af2b2daaa89f]) successfully deleted
I20260929 01:01:26.637071 11270 tablet_bootstrap.cc:492] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: No bootstrap required, opened a new log
I20260929 01:01:26.637276 11270 ts_tablet_manager.cc:1397] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260929 01:01:26.638051 11272 tablet_bootstrap.cc:492] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: No bootstrap required, opened a new log
I20260929 01:01:26.638269 11272 ts_tablet_manager.cc:1397] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260929 01:01:26.638157 11270 raft_consensus.cc:348] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.638405 11270 raft_consensus.cc:374] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.638501 11271 tablet_bootstrap.cc:492] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: No bootstrap required, opened a new log
I20260929 01:01:26.638540 11270 raft_consensus.cc:729] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbadfda961147e4875ab1359c520999, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.638681 11271 ts_tablet_manager.cc:1397] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260929 01:01:26.639351 11272 raft_consensus.cc:348] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.639468 11271 raft_consensus.cc:348] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.639721 11271 raft_consensus.cc:374] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.639734 11272 raft_consensus.cc:374] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:26.639621 11270 consensus_queue.cc:260] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.639829 11271 raft_consensus.cc:729] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874188f28c5449a786ccb82684ed4e56, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.640103 11271 consensus_queue.cc:260] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.640223 11270 ts_tablet_manager.cc:1428] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 01:01:26.640600 11271 ts_tablet_manager.cc:1428] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 01:01:26.641371 11272 raft_consensus.cc:729] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5683e28774f5cbe0621383f97f900, State: Initialized, Role: FOLLOWER
I20260929 01:01:26.641728 11272 consensus_queue.cc:260] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.643045 11272 ts_tablet_manager.cc:1428] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 01:01:26.708395 11277 raft_consensus.cc:482] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:01:26.708638 11277 raft_consensus.cc:504] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.710218 11277 leader_election.cc:288] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:26.733619 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.711114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:42549 (local address 127.3.21.194:37045)
0929 01:01:26.711411 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.711421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.725160 (+ 13739us) server_negotiation.cc:407] Connection header received
0929 01:01:26.725389 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.725399 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.725528 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.725648 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:26.727590 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.728544 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.731167 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.732504 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.732671 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.732746 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.733189 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.733324 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:01:26.734165 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.710540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:37045 (local address 127.3.21.193:42549)
0929 01:01:26.725000 (+ 14460us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.725032 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.725228 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.726757 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.726761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.726790 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:26.727294 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.727308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.728723 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.728732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.730967 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.730985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.732767 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.732774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.732972 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.733015 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.733124 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.733292 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14187,"mutex_wait_us":53,"thread_start_us":13919,"threads_started":1}
I20260929 01:01:26.734586 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.710609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37943 (local address 127.3.21.193:57861)
0929 01:01:26.726476 (+ 15867us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.726523 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.726638 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.727326 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.727332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.727366 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:01:26.727824 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.727838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.729403 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.729409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.731653 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.731673 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.733539 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.733545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.733877 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.733910 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.734020 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.734186 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15740,"mutex_wait_us":29,"thread_start_us":15565,"threads_started":1}
I20260929 01:01:26.734573 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.710685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:57861 (local address 127.3.21.195:37943)
0929 01:01:26.727015 (+ 16330us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.727020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.727038 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:26.727100 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.727104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.727167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.727302 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:01:26.728043 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.729229 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.731872 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.733028 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.733277 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.733355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.734060 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.734201 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16164,"thread_start_us":16015,"threads_started":1}
I20260929 01:01:26.735368 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c1abca10d431dbceafd45bd842166" candidate_uuid: "cbbadfda961147e4875ab1359c520999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900" is_pre_election: true
I20260929 01:01:26.735291 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c1abca10d431dbceafd45bd842166" candidate_uuid: "cbbadfda961147e4875ab1359c520999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "874188f28c5449a786ccb82684ed4e56" is_pre_election: true
I20260929 01:01:26.735637 11196 raft_consensus.cc:2454] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbadfda961147e4875ab1359c520999 in term 0.
I20260929 01:01:26.735766 11061 raft_consensus.cc:2454] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbadfda961147e4875ab1359c520999 in term 0.
I20260929 01:01:26.736238 10847 leader_election.cc:302] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bac5683e28774f5cbe0621383f97f900, cbbadfda961147e4875ab1359c520999; no voters:
I20260929 01:01:26.736791 11277 raft_consensus.cc:2781] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:01:26.736951 11277 raft_consensus.cc:482] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:01:26.737074 11277 raft_consensus.cc:3037] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.739467 11277 raft_consensus.cc:504] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.740307 11277 leader_election.cc:288] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [CANDIDATE]: Term 1 election: Requested vote from peers 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045), bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943)
I20260929 01:01:26.740916 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c1abca10d431dbceafd45bd842166" candidate_uuid: "cbbadfda961147e4875ab1359c520999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "874188f28c5449a786ccb82684ed4e56"
I20260929 01:01:26.741214 11061 raft_consensus.cc:3037] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.743683 11061 raft_consensus.cc:2454] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbadfda961147e4875ab1359c520999 in term 1.
I20260929 01:01:26.744194 10849 leader_election.cc:302] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 874188f28c5449a786ccb82684ed4e56, cbbadfda961147e4875ab1359c520999; no voters:
I20260929 01:01:26.744221 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a7c1abca10d431dbceafd45bd842166" candidate_uuid: "cbbadfda961147e4875ab1359c520999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5683e28774f5cbe0621383f97f900"
I20260929 01:01:26.744460 11196 raft_consensus.cc:3037] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:26.744585 11277 raft_consensus.cc:2781] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:26.745121 11277 raft_consensus.cc:686] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 1 LEADER]: Becoming Leader. State: Replica: cbbadfda961147e4875ab1359c520999, State: Running, Role: LEADER
I20260929 01:01:26.745631 11277 consensus_queue.cc:237] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } }
I20260929 01:01:26.747936 11196 raft_consensus.cc:2454] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbadfda961147e4875ab1359c520999 in term 1.
I20260929 01:01:26.750598 10759 catalog_manager.cc:5495] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbbadfda961147e4875ab1359c520999 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "cbbadfda961147e4875ab1359c520999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbbadfda961147e4875ab1359c520999" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 34755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260929 01:01:26.787408 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.780633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.3.21.193:34755)
0929 01:01:26.781201 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.781205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.781224 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:26.781310 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.781313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.781385 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.781528 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:26.781981 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.783121 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.784542 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.785041 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.785251 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.785929 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.786772 (+ 843us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.786894 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.787040 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.787110 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
I20260929 01:01:26.804723 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.798879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:32935 (local address 127.3.21.194:37045)
0929 01:01:26.799044 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.799049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.799638 (+ 589us) server_negotiation.cc:407] Connection header received
0929 01:01:26.799794 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.799798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.799867 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.799946 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:26.800473 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.801303 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.802755 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.803211 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.803473 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.803937 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.804039 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.804110 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.804414 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.804487 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:01:26.804800 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.798922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:37045 (local address 127.3.21.193:32935)
0929 01:01:26.799530 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.799546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.799675 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.799974 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.799978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.799987 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:01:26.800260 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.800267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.801466 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.801470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.802629 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.802642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.803509 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.803514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.803654 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.803742 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.804255 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.804267 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.804344 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.804464 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":305,"threads_started":1}
I20260929 01:01:26.814000 11196 raft_consensus.cc:1261] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Refusing update from remote peer cbbadfda961147e4875ab1359c520999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.814273 11061 raft_consensus.cc:1261] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Refusing update from remote peer cbbadfda961147e4875ab1359c520999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:01:26.814766 11358 consensus_queue.cc:1035] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5683e28774f5cbe0621383f97f900" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.815284 11277 consensus_queue.cc:1035] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "874188f28c5449a786ccb82684ed4e56" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:01:26.847586 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.842334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:38885 (local address 127.3.21.193:34755)
0929 01:01:26.842552 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.842565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.842708 (+ 143us) server_negotiation.cc:407] Connection header received
0929 01:01:26.842846 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.842850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.842912 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.842986 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:01:26.843402 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.844295 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.845647 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.846189 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.846339 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.846822 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.846937 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.847034 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.847145 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.847258 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:01:26.847645 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.841961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:34755 (local address 127.3.21.194:38885)
0929 01:01:26.842634 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:26.842651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:26.842757 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:26.843015 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:26.843019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:26.843028 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:01:26.843251 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.843258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.844485 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.844489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.845496 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:26.845507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.846359 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.846365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:26.846613 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.846652 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:01:26.847034 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:01:26.847047 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:01:26.847127 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:01:26.847290 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":350,"threads_started":1}
I20260929 01:01:26.896554 10759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:26.896883 10759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:01:26.900835 10759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 579109f95409468784e43f7545c9acc3: Sending DeleteTablet for 3 replicas of tablet 7a7c1abca10d431dbceafd45bd842166
I20260929 01:01:26.902029 11175 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a7c1abca10d431dbceafd45bd842166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:45044
I20260929 01:01:26.902427 10896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a7c1abca10d431dbceafd45bd842166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:56640
I20260929 01:01:26.902689 11035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a7c1abca10d431dbceafd45bd842166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:01:26 UTC) from {username='oryx', principal='oryx/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:42794
I20260929 01:01:26.902830 11332 tablet_replica.cc:330] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: stopping tablet replica
I20260929 01:01:26.903028 11332 raft_consensus.cc:2229] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:26.903291 11332 raft_consensus.cc:2258] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.903395 11331 tablet_replica.cc:330] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: stopping tablet replica
I20260929 01:01:26.903586 11331 raft_consensus.cc:2229] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:01:26.903829 11331 raft_consensus.cc:2258] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.903887 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 10830
I20260929 01:01:26.904162 11330 tablet_replica.cc:330] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: stopping tablet replica
I20260929 01:01:26.904362 11330 raft_consensus.cc:2229] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:01:26.904948 11330 raft_consensus.cc:2258] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:01:26.905133 11332 ts_tablet_manager.cc:1905] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.907045 11331 ts_tablet_manager.cc:1905] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.911600 11330 ts_tablet_manager.cc:1905] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:01:26.916077 11332 ts_tablet_manager.cc:1918] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:01:26.916323 11332 log.cc:1196] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/7a7c1abca10d431dbceafd45bd842166
I20260929 01:01:26.916983 11332 ts_tablet_manager.cc:1939] T 7a7c1abca10d431dbceafd45bd842166 P bac5683e28774f5cbe0621383f97f900: Deleting consensus metadata
I20260929 01:01:26.918828 10744 catalog_manager.cc:4841] TS bac5683e28774f5cbe0621383f97f900 (127.3.21.195:37943): tablet 7a7c1abca10d431dbceafd45bd842166 (table test-table [id=0254fe7b683d4ce8a5f8beb098569967]) successfully deleted
I20260929 01:01:26.921322 11331 ts_tablet_manager.cc:1918] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:01:26.921494 11331 log.cc:1196] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/7a7c1abca10d431dbceafd45bd842166
I20260929 01:01:26.922030 11331 ts_tablet_manager.cc:1939] T 7a7c1abca10d431dbceafd45bd842166 P 874188f28c5449a786ccb82684ed4e56: Deleting consensus metadata
I20260929 01:01:26.923364 10746 catalog_manager.cc:4841] TS 874188f28c5449a786ccb82684ed4e56 (127.3.21.194:37045): tablet 7a7c1abca10d431dbceafd45bd842166 (table test-table [id=0254fe7b683d4ce8a5f8beb098569967]) successfully deleted
I20260929 01:01:26.924733 11330 ts_tablet_manager.cc:1918] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:01:26.924918 11330 log.cc:1196] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/7a7c1abca10d431dbceafd45bd842166
I20260929 01:01:26.925434 11330 ts_tablet_manager.cc:1939] T 7a7c1abca10d431dbceafd45bd842166 P cbbadfda961147e4875ab1359c520999: Deleting consensus metadata
I20260929 01:01:26.926787 10744 catalog_manager.cc:4841] TS cbbadfda961147e4875ab1359c520999 (127.3.21.193:34755): tablet 7a7c1abca10d431dbceafd45bd842166 (table test-table [id=0254fe7b683d4ce8a5f8beb098569967]) successfully deleted
Sep 29 01:01:26 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.193@KRBTEST.COM
I20260929 01:01:26.941632 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:26.905511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.3.21.193:34755)
0929 01:01:26.905743 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:26.905750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:26.923107 (+ 17357us) server_negotiation.cc:407] Connection header received
0929 01:01:26.923715 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:26.923724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:26.923889 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:26.924074 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:26.925897 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.927004 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.928515 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:26.929004 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:26.929161 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:26.934609 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:26.934626 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:26.934643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:26.934710 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:26.938264 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:26.939153 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:26.939168 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:26.939171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:26.939325 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:26.939791 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:26.939795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:26.939801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:26.940205 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:26.940695 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:26.941051 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 01:01:26.941239 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:01:26.942691 10961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:27.058595 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 10830
I20260929 01:01:27.079172 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 10970
Sep 29 01:01:27 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.194@KRBTEST.COM
I20260929 01:01:27.099675 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:27.080204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.3.21.194:37045)
0929 01:01:27.080395 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:27.080399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:27.080416 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:01:27.080502 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:27.080509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:27.080600 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:27.080696 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:27.082897 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:27.084402 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:27.086770 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:27.087233 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:27.087362 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:27.092997 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:27.093014 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:27.093031 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:27.093073 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:27.096248 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:27.097227 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:27.097235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:27.097237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:27.097390 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:27.097768 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:27.097771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:27.097773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:27.098169 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:27.098412 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:27.099037 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 01:01:27.099333 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:01:27.100512 11101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:27.267956 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 10970
I20260929 01:01:27.295161 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11110
Sep 29 01:01:27 dist-test-slave-k0b7 krb5kdc[10714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.21.195@KRBTEST.COM
I20260929 01:01:27.313795 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:27.296106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.3.21.195:37943)
0929 01:01:27.296682 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:27.296687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:27.296705 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:27.296770 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:27.296774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:27.296868 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:27.296968 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:27.298550 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:27.299340 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:27.300697 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:27.301154 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:27.301488 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:27.306463 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:27.306792 (+ 329us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:27.306815 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:27.306867 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:27.310237 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:27.311243 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:27.311251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:27.311254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:27.311492 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:27.311995 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:27.311999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:27.312002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:27.312505 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:27.312806 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:27.313125 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 01:01:27.313343 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":325,"threads_started":1}
I20260929 01:01:27.314787 11241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:27.418685 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11110
I20260929 01:01:27.437347 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 10727
I20260929 01:01:27.438771 10799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:27.557754 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 10727
2026-09-29T01:01:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6295 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:27 dist-test-slave-k0b7 krb5kdc[11380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:27 dist-test-slave-k0b7 krb5kdc[11380](info): set up 2 sockets
Sep 29 01:01:27 dist-test-slave-k0b7 krb5kdc[11380](info): commencing operation
krb5kdc: starting...
W20260929 01:01:28.719561 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:01:28 dist-test-slave-k0b7 krb5kdc[11380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:01:28.975009 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:35547
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:29.332321 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:29.332639 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:29.332751 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:29.340206 11393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:29.340346 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:29.340394 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:29.340444 11393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:29.340493 11393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:29.350721 11393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:29.352083 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:29.353437 11393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:29.366365 11398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:29.366385 11399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:29.366539 11401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:29.367246 11393 server_base.cc:1034] running on GCE node
I20260929 01:01:29.368173 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:29.369817 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:29.371091 11393 hybrid_clock.cc:648] HybridClock initialized: now 1790643689371036 us; error 52 us; skew 500 ppm
I20260929 01:01:29.371608 11393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T01:01:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1897 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:29 dist-test-slave-k0b7 krb5kdc[11407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:29 dist-test-slave-k0b7 krb5kdc[11407](info): set up 2 sockets
Sep 29 01:01:29 dist-test-slave-k0b7 krb5kdc[11407](info): commencing operation
krb5kdc: starting...
W20260929 01:01:30.612133 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:01:30 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:01:30.867514 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34019
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:31.220507 11420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:31.220947 11420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:31.221055 11420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:31.229894 11420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:31.230024 11420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:31.230074 11420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:31.230129 11420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:31.230186 11420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:31.241792 11420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:31.243276 11420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:31.244665 11420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:31.259263 11425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:31.259320 11426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:31.259235 11428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:31.260116 11420 server_base.cc:1034] running on GCE node
I20260929 01:01:31.261099 11420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:31.262957 11420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:31.264210 11420 hybrid_clock.cc:648] HybridClock initialized: now 1790643691264180 us; error 94 us; skew 500 ppm
I20260929 01:01:31.264676 11420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:31 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:31.268628 11420 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:01:31.270699 11420 webserver.cc:460] Webserver started at http://127.3.21.254:36985/ using document root <none> and password file <none>
I20260929 01:01:31.271566 11420 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:31.271693 11420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:31.272110 11420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:31.275870 11420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "6696c8ce570c4eddaff8e0b09577c383"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "8ab1449009ca56eb2c8f3717b6f87476"
server_key_iv: "3e553d4aad4a02fc8b4d7254eb04d27e"
server_key_version: "encryptionkey@0"
I20260929 01:01:31.276875 11420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "6696c8ce570c4eddaff8e0b09577c383"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "8ab1449009ca56eb2c8f3717b6f87476"
server_key_iv: "3e553d4aad4a02fc8b4d7254eb04d27e"
server_key_version: "encryptionkey@0"
I20260929 01:01:31.284092 11420 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 01:01:31.288745 11435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:31.291183 11420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:31.291424 11420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "6696c8ce570c4eddaff8e0b09577c383"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "8ab1449009ca56eb2c8f3717b6f87476"
server_key_iv: "3e553d4aad4a02fc8b4d7254eb04d27e"
server_key_version: "encryptionkey@0"
I20260929 01:01:31.291659 11420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:31.323222 11420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:31.329233 11420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:31.329699 11420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:31.351734 11420 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34019
I20260929 01:01:31.351750 11486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34019 every 8 connection(s)
I20260929 01:01:31.353289 11420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:31.357496 11487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:31.363191 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11420
I20260929 01:01:31.363461 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:31.364203 3159 external_mini_cluster.cc:1428] Setting key a09b6eba23e07cc106a51d3d9cd25e5c
I20260929 01:01:31.370038 11487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Bootstrap starting.
I20260929 01:01:31.373399 11487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:31.374598 11487 log.cc:824] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:31.378024 11487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: No bootstrap required, opened a new log
Sep 29 01:01:31 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:31.386999 11487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:31.387565 11487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:31.387694 11487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6696c8ce570c4eddaff8e0b09577c383, State: Initialized, Role: FOLLOWER
I20260929 01:01:31.388459 11487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:31.388867 11487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:31.389014 11487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:31.389190 11487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:31.390465 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:31.366771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.3.21.254:34019)
0929 01:01:31.367760 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:31.367773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:31.367822 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:01:31.369646 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:31.369675 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:31.370101 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:31.370630 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:31.372165 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:31.373576 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:31.374967 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:31.376187 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:31.376394 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:31.380656 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:31.380664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:31.380687 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:31.380726 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:31.385180 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:31.386162 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:31.386172 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:31.386180 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:31.386386 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:31.386869 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:31.386874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:31.386876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:31.387562 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:31.387935 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:31.388437 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 01:01:31.388947 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":390,"threads_started":1}
I20260929 01:01:31.392575 11487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:31.393260 11487 leader_election.cc:302] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6696c8ce570c4eddaff8e0b09577c383; no voters:
I20260929 01:01:31.394137 11487 leader_election.cc:288] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:31.394387 11492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:31.395246 11492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 LEADER]: Becoming Leader. State: Replica: 6696c8ce570c4eddaff8e0b09577c383, State: Running, Role: LEADER
I20260929 01:01:31.396157 11492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:31.397289 11487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:31.400224 11493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6696c8ce570c4eddaff8e0b09577c383. Latest consensus state: current_term: 1 leader_uuid: "6696c8ce570c4eddaff8e0b09577c383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } } }
I20260929 01:01:31.400882 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6696c8ce570c4eddaff8e0b09577c383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } } }
I20260929 01:01:31.401281 11493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:31.401408 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:31.402869 11499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:31.410131 11499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:31.421689 11499 catalog_manager.cc:1347] Generated new cluster ID: f0bc3453fac448129e266b0a8a67ee3a
I20260929 01:01:31.421873 11499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:31.448179 11499 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:31.450166 11499 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:31.465724 11499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Generated new TSK 0
I20260929 01:01:31.466890 11499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:01:31.539657 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34019
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:31.897185 11513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:31.897497 11513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:31.897626 11513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:31.906276 11513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:31.906437 11513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:31.906798 11513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:31.917405 11513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:31.918890 11513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:31.920310 11513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:31.935102 11521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:31.935266 11518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:31.935240 11519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:31.936901 11513 server_base.cc:1034] running on GCE node
I20260929 01:01:31.937731 11513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:31.938970 11513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:31.940232 11513 hybrid_clock.cc:648] HybridClock initialized: now 1790643691940180 us; error 94 us; skew 500 ppm
I20260929 01:01:31.940591 11513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:31 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:31.944386 11513 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:01:31.946179 11513 webserver.cc:460] Webserver started at http://127.3.21.193:46607/ using document root <none> and password file <none>
I20260929 01:01:31.947041 11513 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:31.947192 11513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:31.947531 11513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:31.950776 11513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "bd9dc7d962a04dfbaa69fb81f6f7a8e2"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "a3be308862f019bf21f3549e45e44d5c"
server_key_iv: "70381d5bd1b00fae5030795eb278089a"
server_key_version: "encryptionkey@0"
I20260929 01:01:31.951602 11513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "bd9dc7d962a04dfbaa69fb81f6f7a8e2"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "a3be308862f019bf21f3549e45e44d5c"
server_key_iv: "70381d5bd1b00fae5030795eb278089a"
server_key_version: "encryptionkey@0"
I20260929 01:01:31.957520 11513 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:01:31.961854 11528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:31.963598 11513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:31.963840 11513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "bd9dc7d962a04dfbaa69fb81f6f7a8e2"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "a3be308862f019bf21f3549e45e44d5c"
server_key_iv: "70381d5bd1b00fae5030795eb278089a"
server_key_version: "encryptionkey@0"
I20260929 01:01:31.964123 11513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:31.997543 11513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:32.003338 11513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:32.003700 11513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:32.004853 11513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:32.006475 11513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:32.006625 11513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:01:32.006803 11513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:32.006901 11513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:32.045305 11513 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:32813
I20260929 01:01:32.045356 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:32813 every 8 connection(s)
I20260929 01:01:32.046736 11513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:32.048099 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11513
I20260929 01:01:32.048292 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:32.049093 3159 external_mini_cluster.cc:1428] Setting key 89941aa248da33950bd97eb46fce6776
Sep 29 01:01:32 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:32.069049 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:32.049752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:60417 (local address 127.3.21.254:34019)
0929 01:01:32.050238 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:32.050242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:32.051493 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 01:01:32.053412 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:32.053417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:32.053509 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:32.053628 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:32.055237 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.055984 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.057694 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.058150 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.058258 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:32.062968 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:32.062974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:32.062978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:32.063012 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:32.065644 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:32.066529 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:32.066535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:32.066538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:32.066675 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:32.067199 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:32.067205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:32.067210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:32.067481 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:32.067640 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:32.068429 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 01:01:32.068651 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":213,"threads_started":1}
I20260929 01:01:32.070175 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:32.050422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:60417)
0929 01:01:32.051319 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:32.051387 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:32.053117 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:32.053880 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:32.053889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:32.054404 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:32.055065 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:32.055082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.056143 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.056147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:32.057521 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:32.057538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.058417 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.058427 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:32.058818 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:32.059634 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:32.059669 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:32.062697 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:32.065854 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:32.065865 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:32.065886 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:32.066368 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:32.066800 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:32.066805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:32.066809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:32.067087 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:32.067615 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:32.067623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:32.068038 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 01:01:32.068316 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 01:01:32.068896 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":309,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:01:32.072078 11641 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34019
I20260929 01:01:32.072490 11641 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:32.073212 11641 heartbeater.cc:511] Master 127.3.21.254:34019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:01:32.075935 11452 ts_manager.cc:194] Registered new tserver with Master: bd9dc7d962a04dfbaa69fb81f6f7a8e2 (127.3.21.193:32813)
I20260929 01:01:32.078125 11452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:60417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:01:32.107175 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34019
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:32.461431 11647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:32.461740 11647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:32.461900 11647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:32.469918 11647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:32.470078 11647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:32.470264 11647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:32.481225 11647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:32.482764 11647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:32.484159 11647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:32.500092 11653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:32.500438 11652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:32.500949 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:32.501232 11647 server_base.cc:1034] running on GCE node
I20260929 01:01:32.502467 11647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:32.503921 11647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:32.505168 11647 hybrid_clock.cc:648] HybridClock initialized: now 1790643692505131 us; error 126 us; skew 500 ppm
I20260929 01:01:32.505604 11647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:32 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:32.509495 11647 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:01:32.511682 11647 webserver.cc:460] Webserver started at http://127.3.21.194:41483/ using document root <none> and password file <none>
I20260929 01:01:32.512499 11647 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:32.512660 11647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:32.513020 11647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:32.516292 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "e5908453054f4ac4aaeb0a983aa500b5"
format_stamp: "Formatted at 2026-09-29 01:01:32 on dist-test-slave-k0b7"
server_key: "11ae2bd847f1256cfabebb4938cae597"
server_key_iv: "adf2e7c9e225580d97ee0cbb19464c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:32.517150 11647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "e5908453054f4ac4aaeb0a983aa500b5"
format_stamp: "Formatted at 2026-09-29 01:01:32 on dist-test-slave-k0b7"
server_key: "11ae2bd847f1256cfabebb4938cae597"
server_key_iv: "adf2e7c9e225580d97ee0cbb19464c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:32.523085 11647 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:01:32.527079 11662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:32.528638 11647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:01:32.528877 11647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "e5908453054f4ac4aaeb0a983aa500b5"
format_stamp: "Formatted at 2026-09-29 01:01:32 on dist-test-slave-k0b7"
server_key: "11ae2bd847f1256cfabebb4938cae597"
server_key_iv: "adf2e7c9e225580d97ee0cbb19464c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:32.529100 11647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:32.571929 11647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:32.577483 11647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:32.577854 11647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:32.579070 11647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:32.580755 11647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:32.580874 11647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:32.581063 11647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:32.581144 11647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:32.624784 11647 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:44295
I20260929 01:01:32.624837 11774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:44295 every 8 connection(s)
I20260929 01:01:32.626251 11647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:32.633222 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11647
I20260929 01:01:32.633419 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:32.633963 3159 external_mini_cluster.cc:1428] Setting key 3b8401f26ddb0f46d094916312e0cfbd
Sep 29 01:01:32 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:32.652009 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:32.629754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:43117 (local address 127.3.21.254:34019)
0929 01:01:32.630297 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:32.630303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:32.631213 (+ 910us) server_negotiation.cc:407] Connection header received
0929 01:01:32.633360 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:32.633367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:32.633478 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:32.633636 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:32.635609 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.636693 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.639624 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.640280 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.640419 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:32.645125 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:32.645131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:32.645134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:32.645176 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:32.647927 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:32.649111 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:32.649115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:32.649118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:32.649358 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:32.649975 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:32.649978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:32.649981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:32.650313 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:32.650449 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:32.651546 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0929 01:01:32.651731 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":319,"threads_started":1}
I20260929 01:01:32.653746 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:32.630269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:43117)
0929 01:01:32.631041 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:32.631080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:32.633002 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:32.633902 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:32.633913 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:32.634708 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:32.635431 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:32.635450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.636894 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.636902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:32.639363 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:32.639384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:32.640563 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:32.640572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:32.640949 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:32.641817 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:32.641852 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:32.644766 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:32.648107 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:32.648122 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:32.648147 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:32.648921 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:32.649501 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:32.649510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:32.649517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:32.649799 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:32.650475 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:32.650524 (+ 49us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:32.651044 (+ 520us) client_negotiation.cc:769] Sending connection context
0929 01:01:32.651404 (+ 360us) client_negotiation.cc:240] Negotiation successful
0929 01:01:32.652069 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":334,"threads_started":1}
I20260929 01:01:32.655738 11775 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34019
I20260929 01:01:32.656113 11775 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:32.656893 11775 heartbeater.cc:511] Master 127.3.21.254:34019 requested a full tablet report, sending...
I20260929 01:01:32.658440 11452 ts_manager.cc:194] Registered new tserver with Master: e5908453054f4ac4aaeb0a983aa500b5 (127.3.21.194:44295)
I20260929 01:01:32.659317 11452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:43117
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:01:32.695192 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34019
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:33.049687 11781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:33.050091 11781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:33.050254 11781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:33.058696 11781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:33.058816 11781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:33.058935 11781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:33.069253 11781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:33.070623 11781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:33.071902 11781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:01:33.081588 11641 heartbeater.cc:503] Master 127.3.21.254:34019 was elected leader, sending a full tablet report...
W20260929 01:01:33.086424 11786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:33.086498 11787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:33.086452 11789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:33.087241 11781 server_base.cc:1034] running on GCE node
I20260929 01:01:33.087831 11781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:33.088904 11781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:33.090118 11781 hybrid_clock.cc:648] HybridClock initialized: now 1790643693090067 us; error 53 us; skew 500 ppm
I20260929 01:01:33.090411 11781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:33 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:33.093981 11781 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:01:33.095829 11781 webserver.cc:460] Webserver started at http://127.3.21.195:45467/ using document root <none> and password file <none>
I20260929 01:01:33.096572 11781 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:33.096740 11781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:33.097021 11781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:33.100459 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "1523cb6f85244e2885092adb368e4cfb"
format_stamp: "Formatted at 2026-09-29 01:01:33 on dist-test-slave-k0b7"
server_key: "e3833e34319ceb0a363317ed3a3f7ac9"
server_key_iv: "29e1d6d736902c38c54fdeb5bf3478a1"
server_key_version: "encryptionkey@0"
I20260929 01:01:33.101331 11781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "1523cb6f85244e2885092adb368e4cfb"
format_stamp: "Formatted at 2026-09-29 01:01:33 on dist-test-slave-k0b7"
server_key: "e3833e34319ceb0a363317ed3a3f7ac9"
server_key_iv: "29e1d6d736902c38c54fdeb5bf3478a1"
server_key_version: "encryptionkey@0"
I20260929 01:01:33.107486 11781 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 01:01:33.113940 11796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:33.116503 11781 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:01:33.116757 11781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "1523cb6f85244e2885092adb368e4cfb"
format_stamp: "Formatted at 2026-09-29 01:01:33 on dist-test-slave-k0b7"
server_key: "e3833e34319ceb0a363317ed3a3f7ac9"
server_key_iv: "29e1d6d736902c38c54fdeb5bf3478a1"
server_key_version: "encryptionkey@0"
I20260929 01:01:33.117031 11781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:33.141939 11781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:33.147891 11781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:33.148212 11781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:33.149281 11781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:33.150745 11781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:33.150827 11781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:33.150959 11781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:33.151010 11781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:33.186733 11781 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:36163
I20260929 01:01:33.186847 11908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:36163 every 8 connection(s)
I20260929 01:01:33.188042 11781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:33.190907 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11781
I20260929 01:01:33.191123 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:01:33.191668 3159 external_mini_cluster.cc:1428] Setting key c9a9141e1bb6c1201c193dc7101550e3
Sep 29 01:01:33 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:33.212523 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:33.191066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:53809 (local address 127.3.21.254:34019)
0929 01:01:33.191567 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:33.191573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:33.192792 (+ 1219us) server_negotiation.cc:407] Connection header received
0929 01:01:33.195489 (+ 2697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:33.195493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:33.195559 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:33.195652 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:33.197815 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.198912 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.200605 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.200986 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.201090 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:33.205523 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:33.205529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:33.205533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:33.205570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:33.209095 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.209975 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.209980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.209982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.210116 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.210645 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.210648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.210650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.210931 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:33.211053 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:33.212033 (+ 980us) server_negotiation.cc:299] Negotiation successful
0929 01:01:33.212219 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260929 01:01:33.213613 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:33.191656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:53809)
0929 01:01:33.192570 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:33.192612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:33.195115 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:33.195961 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:33.195970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:33.196707 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:33.197595 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:33.197624 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.199085 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.199089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:33.200463 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:33.200475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.201252 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.201260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:33.201488 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:33.202346 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:33.202378 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:33.205102 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:33.209247 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:33.209256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:33.209272 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:33.209821 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:33.210218 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:33.210222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:33.210224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:33.210468 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:33.211080 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:33.211087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:33.211542 (+ 455us) client_negotiation.cc:769] Sending connection context
0929 01:01:33.211914 (+ 372us) client_negotiation.cc:240] Negotiation successful
0929 01:01:33.212348 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":343,"threads_started":1}
I20260929 01:01:33.215440 11909 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34019
I20260929 01:01:33.215756 11909 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:33.216642 11909 heartbeater.cc:511] Master 127.3.21.254:34019 requested a full tablet report, sending...
I20260929 01:01:33.218544 11452 ts_manager.cc:194] Registered new tserver with Master: 1523cb6f85244e2885092adb368e4cfb (127.3.21.195:36163)
I20260929 01:01:33.219652 11452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:53809
I20260929 01:01:33.221016 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab.
Entry for principal oryx/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab.
WARNING: no policy specified for oryx/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab.
Entry for principal oryx/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab.
WARNING: no policy specified for oryx/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab.
Entry for principal oryx/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab.
I20260929 01:01:33.383625 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11513
Sep 29 01:01:33 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:01:33.402740 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:33.385131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.3.21.193:32813)
0929 01:01:33.385934 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:33.385941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:33.385970 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:01:33.386029 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:33.386035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:33.386263 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:33.386411 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:33.387909 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.389056 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.390394 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.391096 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.391205 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:33.395936 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:33.395962 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:33.395986 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:33.396029 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:33.399496 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.400362 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.400371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.400373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.400514 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.400909 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.400913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.400916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.401335 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:33.401602 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:33.402017 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 01:01:33.402233 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":488,"threads_started":1}
I20260929 01:01:33.404181 11636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:33.490780 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11513
I20260929 01:01:33.508387 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11647
Sep 29 01:01:33 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:01:33.526700 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:33.509375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.3.21.194:44295)
0929 01:01:33.509979 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:33.509986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:33.510011 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:01:33.510070 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:33.510076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:33.510288 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:33.510538 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:33.511817 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.512914 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.514264 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.514946 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.515096 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:33.519938 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:33.519953 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:33.519970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:33.520013 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:33.523507 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.524298 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.524305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.524308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.524435 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.524821 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.524827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.524830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.525209 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:33.525527 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:33.525859 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 01:01:33.526204 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":338,"threads_started":1}
I20260929 01:01:33.528083 11770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:33.614594 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11647
I20260929 01:01:33.632273 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11781
Sep 29 01:01:33 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:01:33.651737 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:33.633439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.3.21.195:36163)
0929 01:01:33.633995 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:33.634004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:33.634051 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:01:33.634118 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:33.634124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:33.634353 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:33.634582 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:33.636039 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.637065 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.638669 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:33.639359 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:33.639466 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:33.644569 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:33.644587 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:33.644605 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:33.644651 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:33.648181 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.649075 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.649085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.649093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.649324 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:33.649736 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:33.649741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:33.649744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:33.650163 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:33.650417 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:33.650998 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 01:01:33.651335 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":270,"threads_started":1}
I20260929 01:01:33.653270 11904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:33.739732 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11781
I20260929 01:01:33.756150 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11420
I20260929 01:01:33.757786 11482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:33.865211 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11420
I20260929 01:01:33.887130 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34019
--webserver_interface=127.3.21.254
--webserver_port=36985
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:34.240455 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:34.240761 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:34.240914 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:34.248664 11927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:34.248804 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:34.248876 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:34.248970 11927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:34.249037 11927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:34.259600 11927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:34.261071 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:34.262444 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:34.275903 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:34.275849 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:34.275827 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:34.277034 11927 server_base.cc:1034] running on GCE node
I20260929 01:01:34.277977 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:34.280153 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:34.281387 11927 hybrid_clock.cc:648] HybridClock initialized: now 1790643694281322 us; error 52 us; skew 500 ppm
I20260929 01:01:34.281772 11927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643694, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:34.285583 11927 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:01:34.287626 11927 webserver.cc:460] Webserver started at http://127.3.21.254:36985/ using document root <none> and password file <none>
I20260929 01:01:34.288437 11927 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:34.288594 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:34.295502 11927 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:34.298754 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:34.300261 11927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:01:34.300473 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "6696c8ce570c4eddaff8e0b09577c383"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "8ab1449009ca56eb2c8f3717b6f87476"
server_key_iv: "3e553d4aad4a02fc8b4d7254eb04d27e"
server_key_version: "encryptionkey@0"
I20260929 01:01:34.301854 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:34.342608 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:34.348299 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:34.348654 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:34.369285 11927 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34019
I20260929 01:01:34.369331 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34019 every 8 connection(s)
I20260929 01:01:34.370929 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:34.373070 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11927
I20260929 01:01:34.373834 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:32813
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:34019
--webserver_port=46607
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:01:34.379563 11994 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:01:34.384337 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Bootstrap starting.
I20260929 01:01:34.407199 11994 log.cc:824] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:34.418279 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:01:34.419010 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Bootstrap complete.
I20260929 01:01:34.429849 11994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:34.430996 11994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6696c8ce570c4eddaff8e0b09577c383, State: Initialized, Role: FOLLOWER
I20260929 01:01:34.431704 11994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:34.432020 11994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:34.432189 11994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:34.432399 11994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:01:34.437157 11994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:34.437732 11994 leader_election.cc:302] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6696c8ce570c4eddaff8e0b09577c383; no voters:
I20260929 01:01:34.438455 11994 leader_election.cc:288] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:01:34.438618 11998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:01:34.439882 11998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [term 2 LEADER]: Becoming Leader. State: Replica: 6696c8ce570c4eddaff8e0b09577c383, State: Running, Role: LEADER
I20260929 01:01:34.440519 11998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } }
I20260929 01:01:34.440842 11994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:34.443001 12000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6696c8ce570c4eddaff8e0b09577c383. Latest consensus state: current_term: 2 leader_uuid: "6696c8ce570c4eddaff8e0b09577c383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } } }
I20260929 01:01:34.443409 12000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:34.443974 11999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6696c8ce570c4eddaff8e0b09577c383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6696c8ce570c4eddaff8e0b09577c383" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34019 } } }
I20260929 01:01:34.444320 11999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:34.454954 12004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:34.461776 12004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:34.463423 12004 catalog_manager.cc:1259] Loaded cluster ID: f0bc3453fac448129e266b0a8a67ee3a
I20260929 01:01:34.463655 12004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:34.468809 12004 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:34.470805 12004 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6696c8ce570c4eddaff8e0b09577c383: Loaded TSK: 0
I20260929 01:01:34.472664 12004 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:01:34.742345 11996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:34.742758 11996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:34.742924 11996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:34.750368 11996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:34.750569 11996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:34.750765 11996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:34.761121 11996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.193.keytab
--principal=oryx/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:32813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=46607
--tserver_master_addrs=127.3.21.254:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:34.763005 11996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:34.765147 11996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:34.783941 12023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:34.784248 12020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:34.784514 12021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:34.784842 11996 server_base.cc:1034] running on GCE node
I20260929 01:01:34.785532 11996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:34.786687 11996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:34.788010 11996 hybrid_clock.cc:648] HybridClock initialized: now 1790643694787957 us; error 45 us; skew 500 ppm
I20260929 01:01:34.788394 11996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643694, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:34.800834 11996 init.cc:377] Logged in from keytab as oryx/127.3.21.193@KRBTEST.COM (short username oryx)
I20260929 01:01:34.802942 11996 webserver.cc:460] Webserver started at http://127.3.21.193:46607/ using document root <none> and password file <none>
I20260929 01:01:34.803704 11996 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:34.803879 11996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:34.809991 11996 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:01:34.813081 12030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:34.814729 11996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:34.814932 11996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "bd9dc7d962a04dfbaa69fb81f6f7a8e2"
format_stamp: "Formatted at 2026-09-29 01:01:31 on dist-test-slave-k0b7"
server_key: "a3be308862f019bf21f3549e45e44d5c"
server_key_iv: "70381d5bd1b00fae5030795eb278089a"
server_key_version: "encryptionkey@0"
I20260929 01:01:34.815989 11996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:34.842298 11996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:34.847254 11996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:34.847564 11996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:34.848644 11996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:34.850106 11996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:34.850196 11996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:34.850304 11996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:34.850409 11996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:34.888386 11996 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:32813
I20260929 01:01:34.888438 12142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:32813 every 8 connection(s)
I20260929 01:01:34.889889 11996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:34.891191 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 11996
I20260929 01:01:34.892006 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:44295
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:34019
--webserver_port=41483
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:34.916950 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:34.893763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:49207)
0929 01:01:34.895335 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:34.895390 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:34.898671 (+ 3281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:34.900647 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:34.900659 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:34.901604 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:34.902470 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:34.902538 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.904692 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.904702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:34.907383 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:34.907407 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.908690 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.908701 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:34.909100 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:34.909924 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:34.909965 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:34.915418 (+ 5453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":912,"threads_started":1}
W20260929 01:01:34.916980 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:34.893704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:49207 (local address 127.3.21.254:34019)
0929 01:01:34.894634 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:34.894653 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:34.895576 (+ 923us) server_negotiation.cc:407] Connection header received
0929 01:01:34.899538 (+ 3962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:34.899554 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:34.899990 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:34.900468 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:34.902881 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.904461 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.907597 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.908405 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.908571 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:34.915836 (+ 7265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:49207: BlockingRecv error: recv got EOF from 127.3.21.193:49207 (error 108)
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":342,"threads_started":1}
W20260929 01:01:34.917937 12143 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:34019 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.21.254:34019: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:34.928498 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:34.918712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:49101)
0929 01:01:34.918972 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:34.918990 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:34.919093 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:34.919529 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:34.919534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:34.920070 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:34.920245 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:34.920255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.921902 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.921906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:34.923140 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:34.923154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.924179 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.924184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:34.924323 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:34.925019 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:34.925037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:34.928286 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 01:01:34.929024 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:34.918865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:49101 (local address 127.3.21.254:34019)
0929 01:01:34.919095 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:34.919103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:34.919160 (+ 57us) server_negotiation.cc:407] Connection header received
0929 01:01:34.919233 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:34.919242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:34.919343 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:34.919466 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:34.920411 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.921732 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.923286 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.923992 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.924128 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:34.928575 (+ 4447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:49101: BlockingRecv error: recv got EOF from 127.3.21.193:49101 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:34 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:34.939371 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:34.929318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:35417)
0929 01:01:34.929553 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:34.929573 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:34.929690 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:34.930062 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:34.930068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:34.930519 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:34.930737 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:34.930750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.931997 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.932003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:34.933554 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:34.933571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.934410 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.934419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:34.934647 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:34.935387 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:34.935411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:34.939118 (+ 3707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 01:01:34.939711 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:34.929603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:35417 (local address 127.3.21.254:34019)
0929 01:01:34.929766 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:34.929770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:34.929786 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:34.929833 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:34.929836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:34.929904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:34.929986 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:34.930929 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.931831 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.933722 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:34.934234 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:34.934372 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:34.939431 (+ 5059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:35417: BlockingRecv error: recv got EOF from 127.3.21.193:35417 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 01:01:34.939823 12143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:01:35.273761 12147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:35.274127 12147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:35.274313 12147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:35.282222 12147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:35.282400 12147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:35.282660 12147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:35.293049 12147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.194.keytab
--principal=oryx/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:44295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=41483
--tserver_master_addrs=127.3.21.254:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:35.294541 12147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:35.295915 12147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:35.310979 12155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:35.311034 12153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:35.311012 12152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:35.312044 12147 server_base.cc:1034] running on GCE node
I20260929 01:01:35.312826 12147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:35.313846 12147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:35.315099 12147 hybrid_clock.cc:648] HybridClock initialized: now 1790643695315002 us; error 93 us; skew 500 ppm
I20260929 01:01:35.315474 12147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643695, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:35.319466 12147 init.cc:377] Logged in from keytab as oryx/127.3.21.194@KRBTEST.COM (short username oryx)
I20260929 01:01:35.321555 12147 webserver.cc:460] Webserver started at http://127.3.21.194:41483/ using document root <none> and password file <none>
I20260929 01:01:35.322355 12147 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:35.322579 12147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:35.328927 12147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:35.332559 12162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:35.334462 12147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:01:35.334782 12147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "e5908453054f4ac4aaeb0a983aa500b5"
format_stamp: "Formatted at 2026-09-29 01:01:32 on dist-test-slave-k0b7"
server_key: "11ae2bd847f1256cfabebb4938cae597"
server_key_iv: "adf2e7c9e225580d97ee0cbb19464c61"
server_key_version: "encryptionkey@0"
I20260929 01:01:35.335875 12147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:35.353590 12147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:35.358889 12147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:35.359272 12147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:35.360423 12147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:35.362030 12147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:35.362164 12147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:35.362315 12147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:35.362426 12147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:35.400939 12147 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:44295
I20260929 01:01:35.401002 12274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:44295 every 8 connection(s)
I20260929 01:01:35.402700 12147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:35.404191 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12147
I20260929 01:01:35.405040 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:36163
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:34019
--webserver_port=45467
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:01:35.423648 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.406028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:43559 (local address 127.3.21.254:34019)
0929 01:01:35.406255 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.406269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.407944 (+ 1675us) server_negotiation.cc:407] Connection header received
0929 01:01:35.411106 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.411115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.411293 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.411403 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.413446 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.414715 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.416534 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.417347 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.417522 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.423392 (+ 5870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:43559: BlockingRecv error: recv got EOF from 127.3.21.194:43559 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:01:35.424446 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.406810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:43559)
0929 01:01:35.407738 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.407792 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.410590 (+ 2798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.411716 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.411731 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.412583 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.413216 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.413234 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.414945 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.414950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.416351 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.416366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.417626 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.417632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.417859 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.418717 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.418741 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.423152 (+ 4411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":357,"threads_started":1}
W20260929 01:01:35.425493 12275 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:34019 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.21.254:34019: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:35.435385 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.426257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:47189)
0929 01:01:35.426583 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.426601 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.426707 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.427141 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.427145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.427510 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.427655 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.427663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.429320 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.429324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.430241 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.430254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.431461 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.431465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.431604 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.432269 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.432293 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.435171 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149}
W20260929 01:01:35.435814 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.426353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:47189 (local address 127.3.21.254:34019)
0929 01:01:35.426649 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.426659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.426688 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:01:35.426847 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.426856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.426956 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.427091 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.427829 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.429154 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.430403 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.431279 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.431472 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.435516 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:47189: BlockingRecv error: recv got EOF from 127.3.21.194:47189 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:35.445551 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.436395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:43859)
0929 01:01:35.436560 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.436586 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.436678 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.437075 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.437078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.437394 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.437549 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.437557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.439299 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.439302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.440532 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.440545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.441378 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.441384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.441596 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.442300 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.442318 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.445238 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 01:01:35.445848 12275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:01:35.445684 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.436246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:43859 (local address 127.3.21.254:34019)
0929 01:01:35.436437 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.436444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.436656 (+ 212us) server_negotiation.cc:407] Connection header received
0929 01:01:35.436797 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.436803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.436885 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.437002 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.437693 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.439118 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.440673 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.441171 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.441357 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.445462 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:43859: BlockingRecv error: recv got EOF from 127.3.21.194:43859 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 01:01:35.758853 12278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:35.759204 12278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:35.759464 12278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:35.767659 12278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:35.767827 12278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:35.768002 12278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:35.778154 12278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/krb5kdc/oryx_127.3.21.195.keytab
--principal=oryx/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=45467
--tserver_master_addrs=127.3.21.254:34019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:35.779629 12278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:35.780953 12278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:35.795488 12286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:35.795531 12284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:35.795871 12283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:35.796315 12278 server_base.cc:1034] running on GCE node
I20260929 01:01:35.796980 12278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:35.798048 12278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:35.799325 12278 hybrid_clock.cc:648] HybridClock initialized: now 1790643695799230 us; error 110 us; skew 500 ppm
I20260929 01:01:35.799680 12278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643695, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:35.803018 12278 init.cc:377] Logged in from keytab as oryx/127.3.21.195@KRBTEST.COM (short username oryx)
I20260929 01:01:35.804953 12278 webserver.cc:460] Webserver started at http://127.3.21.195:45467/ using document root <none> and password file <none>
I20260929 01:01:35.805742 12278 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:35.805913 12278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:35.812695 12278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:01:35.815872 12293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:35.817826 12278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:35.818058 12278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "1523cb6f85244e2885092adb368e4cfb"
format_stamp: "Formatted at 2026-09-29 01:01:33 on dist-test-slave-k0b7"
server_key: "e3833e34319ceb0a363317ed3a3f7ac9"
server_key_iv: "29e1d6d736902c38c54fdeb5bf3478a1"
server_key_version: "encryptionkey@0"
I20260929 01:01:35.819273 12278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:35.859472 12278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:35.864737 12278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:35.865084 12278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:35.866212 12278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:35.867749 12278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:35.867895 12278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:35.868080 12278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:35.868181 12278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:35.908437 12278 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:36163
I20260929 01:01:35.908547 12405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:36163 every 8 connection(s)
I20260929 01:01:35.910176 12278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:35.913893 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12278
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:01:35.929339 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.913545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:42829 (local address 127.3.21.254:34019)
0929 01:01:35.913813 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.913820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.915054 (+ 1234us) server_negotiation.cc:407] Connection header received
0929 01:01:35.917134 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.917140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.917264 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.917374 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.919666 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.920611 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.922713 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.923213 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.923420 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.929031 (+ 5611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:42829: BlockingRecv error: recv got EOF from 127.3.21.195:42829 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 01:01:35.930038 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.913917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:42829)
0929 01:01:35.914894 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.914940 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.916838 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.917881 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.917896 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.918676 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.919433 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.919464 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.920876 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.920881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.922555 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.922572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.923521 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.923529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.923881 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.924821 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.924867 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.928805 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":382,"threads_started":1}
W20260929 01:01:35.930779 12406 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:34019 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.21.254:34019: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
I20260929 01:01:35.934667 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.915004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.3.21.254:34019)
0929 01:01:35.918892 (+ 3888us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.918896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.918911 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:35.918953 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.918956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.919032 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.919120 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.920873 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.921661 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.923271 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.923846 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.923978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.927094 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:35.927103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:35.927122 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:35.927164 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:35.930722 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:35.931670 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:35.931679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:35.931688 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:35.931918 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:35.932426 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:35.932432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:35.932435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:35.933050 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:35.933432 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:35.933770 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 01:01:35.934135 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3779,"thread_start_us":3655,"threads_started":1}
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:35.941001 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.931596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:47093)
0929 01:01:35.931814 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.931834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.931936 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.932310 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.932315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.932751 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.932973 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.932985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.934457 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.934462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.935533 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.935545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.936434 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.936439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.936605 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.937245 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.937270 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.940644 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 01:01:35.941269 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.931686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:47093 (local address 127.3.21.254:34019)
0929 01:01:35.931889 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.931893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.931909 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:35.932061 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.932067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.932164 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.932267 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.933159 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.934318 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.935669 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.936281 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.936400 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.940841 (+ 4441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:47093: BlockingRecv error: recv got EOF from 127.3.21.195:47093 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:35 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:35.950516 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.940474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:35131)
0929 01:01:35.941219 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.941235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.941345 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.942176 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.942183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.942772 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.943046 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.943062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.944105 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.944112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.945130 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.945142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.945865 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.945884 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.946046 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.946722 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.946741 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.950140 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":474,"threads_started":1}
I20260929 01:01:35.951033 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:35.941673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:38677)
0929 01:01:35.942314 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:35.942328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:35.942428 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:35.943009 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:35.943012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:35.943429 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:35.943629 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.943637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.944715 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.944720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.945701 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:35.945712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.946756 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.946762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:35.946923 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.947483 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:35.947500 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:35.950729 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":281,"threads_started":1}
W20260929 01:01:35.951328 12406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:01:35.951442 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.941759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:38677 (local address 127.3.21.254:34019)
0929 01:01:35.942684 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.942689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.942708 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:35.942766 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.942771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.942843 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.942949 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.943880 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.944572 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.945849 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.946596 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.946716 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.951040 (+ 4324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:38677: BlockingRecv error: recv got EOF from 127.3.21.195:38677 (error 108)
Metrics: {"server-negotiator.queue_time_us":785}
W20260929 01:01:35.951717 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:35.940637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:35131 (local address 127.3.21.254:34019)
0929 01:01:35.941691 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:35.941696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:35.941714 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:35.941768 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:35.941784 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:35.941869 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:35.941993 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:35.943205 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.943954 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.945286 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:35.945676 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:35.945803 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:35.951505 (+ 5702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:35131: BlockingRecv error: recv got EOF from 127.3.21.193:35131 (error 108)
Metrics: {"server-negotiator.queue_time_us":941}
Sep 29 01:01:36 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:36 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:36.456940 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:36.446580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:54673)
0929 01:01:36.447275 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:36.447298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:36.447456 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:36.447898 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:36.447902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:36.448425 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:36.448647 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:36.448659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.449955 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.449959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:36.451352 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:36.451368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.452577 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.452582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:36.452706 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:36.453375 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:36.453393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:36.456601 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":335,"threads_started":1}
W20260929 01:01:36.457191 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:36.446798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:54673 (local address 127.3.21.254:34019)
0929 01:01:36.447032 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:36.447037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:36.447424 (+ 387us) server_negotiation.cc:407] Connection header received
0929 01:01:36.447588 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:36.447592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:36.447712 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:36.447811 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:36.448854 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.449744 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.451526 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.452343 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.452520 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:36.456973 (+ 4453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:54673: BlockingRecv error: recv got EOF from 127.3.21.194:54673 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 01:01:36 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:36 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:36 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:36 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:36.962937 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:36.952036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:49101)
0929 01:01:36.953323 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:36.953346 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:36.953564 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:36.954182 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:36.954189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:36.954821 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:36.955063 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:36.955077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.956537 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.956544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:36.957637 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:36.957649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.958460 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.958466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:36.958684 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:36.959357 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:36.959383 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:36.962624 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":1003,"threads_started":1}
W20260929 01:01:36.963207 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:36.952031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:49101 (local address 127.3.21.254:34019)
0929 01:01:36.952448 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:36.952458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:36.953447 (+ 989us) server_negotiation.cc:407] Connection header received
0929 01:01:36.953735 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:36.953740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:36.953843 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:36.953963 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:36.955312 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.956398 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.957787 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.958258 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.958392 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:36.962889 (+ 4497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:49101: BlockingRecv error: recv got EOF from 127.3.21.195:49101 (error 108)
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":14976}
W20260929 01:01:36.963503 12416 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:36.952031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:51369 (local address 127.3.21.254:34019)
0929 01:01:36.953634 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:36.953641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:36.953659 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:36.953731 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:36.953736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:36.953844 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:36.953958 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:36.955244 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.956408 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.957944 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.958424 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.958623 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:36.963201 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:51369: BlockingRecv error: recv got EOF from 127.3.21.193:51369 (error 108)
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":1470,"spinlock_wait_cycles":18048,"thread_start_us":1253,"threads_started":1}
I20260929 01:01:36.963570 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:36.951848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:51369)
0929 01:01:36.952955 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:36.952982 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:36.953172 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:36.954049 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:36.954055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:36.954815 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:36.955082 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:36.955096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.956589 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.956602 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:36.957760 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:36.957774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:36.958664 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:36.958671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:36.958867 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:36.959553 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:36.959574 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:36.962933 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":781,"threads_started":1}
Sep 29 01:01:37 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:37 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:01:37.468616 12416 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:37.458202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:60249 (local address 127.3.21.254:34019)
0929 01:01:37.458408 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:37.458415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:37.459013 (+ 598us) server_negotiation.cc:407] Connection header received
0929 01:01:37.459150 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:37.459155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:37.459238 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:37.459350 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:37.460123 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.461093 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.462639 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.463072 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.463181 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:37.468381 (+ 5200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:60249: BlockingRecv error: recv got EOF from 127.3.21.194:60249 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:01:37.468638 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:37.458114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:60249)
0929 01:01:37.458915 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:37.458933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:37.459048 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:37.459409 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:37.459414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:37.459859 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:37.460006 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:37.460016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.461279 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.461285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:37.462420 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:37.462435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.463226 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.463230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:37.463481 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:37.464413 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:37.464433 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:37.468089 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":444,"threads_started":1}
Sep 29 01:01:37 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:37 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:37 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:37 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:37.974555 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:37.964209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:58339)
0929 01:01:37.965135 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:37.965164 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:37.965375 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:37.965941 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:37.965947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:37.966402 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:37.966636 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:37.966648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.967715 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.967723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:37.968639 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:37.968650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.969444 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.969449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:37.969571 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:37.970325 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:37.970349 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:37.974187 (+ 3838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":600,"threads_started":1}
I20260929 01:01:37.974854 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:37.964470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:41383)
0929 01:01:37.965457 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:37.965481 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:37.965612 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:37.966068 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:37.966074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:37.966438 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:37.966741 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:37.966761 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.967974 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.967981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:37.969020 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:37.969036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.969934 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.969941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:37.970092 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:37.970761 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:37.970791 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:37.974453 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":682,"threads_started":1}
W20260929 01:01:37.974882 12416 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:37.964596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:58339 (local address 127.3.21.254:34019)
0929 01:01:37.964838 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:37.964846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:37.965243 (+ 397us) server_negotiation.cc:407] Connection header received
0929 01:01:37.965553 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:37.965559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:37.965646 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:37.965715 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:37.966782 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.967581 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.968783 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.969276 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.969394 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:37.974541 (+ 5147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:58339: BlockingRecv error: recv got EOF from 127.3.21.195:58339 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 01:01:37.975209 12420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:37.964963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:41383 (local address 127.3.21.254:34019)
0929 01:01:37.965725 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:37.965729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:37.965744 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:37.965780 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:37.965784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:37.965875 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:37.965973 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:37.966934 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.967826 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.969171 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:37.969760 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:37.969856 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:37.974901 (+ 5045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:41383: BlockingRecv error: recv got EOF from 127.3.21.193:41383 (error 108)
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":501,"threads_started":1}
Sep 29 01:01:38 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:38 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:38.479156 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:38.469771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:38559)
0929 01:01:38.470470 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:38.470529 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:38.470654 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:38.471052 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:38.471056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:38.471405 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:38.471579 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:38.471587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.472689 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.472693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:38.473726 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:38.473740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.474632 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.474637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:38.474772 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:38.475537 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:38.475555 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:38.478859 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"spinlock_wait_cycles":130304,"thread_start_us":334,"threads_started":1}
W20260929 01:01:38.479444 12420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:38.469856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:38559 (local address 127.3.21.254:34019)
0929 01:01:38.470024 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:38.470028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:38.470618 (+ 590us) server_negotiation.cc:407] Connection header received
0929 01:01:38.470789 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:38.470793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:38.470867 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:38.470956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:38.471722 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.472538 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.473901 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.474415 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.474638 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:38.479137 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:38559: BlockingRecv error: recv got EOF from 127.3.21.194:38559 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 01:01:38 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:38 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:38 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:38 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:38.986927 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:38.976145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:60665)
0929 01:01:38.976670 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:38.976694 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:38.976917 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:38.977746 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:38.977749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:38.978131 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:38.978297 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:38.978306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.980064 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.980068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:38.980977 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:38.980988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.981962 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.981968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:38.982170 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:38.982923 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:38.982943 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:38.986424 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":279,"threads_started":1}
W20260929 01:01:38.987068 12424 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:38.976545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:60665 (local address 127.3.21.254:34019)
0929 01:01:38.977444 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:38.977448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:38.977466 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:01:38.977506 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:38.977509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:38.977617 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:38.977726 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:38.978536 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.979863 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.981128 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.981798 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.981969 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:38.986732 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:60665: BlockingRecv error: recv got EOF from 127.3.21.195:60665 (error 108)
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":771,"spinlock_wait_cycles":768,"thread_start_us":563,"threads_started":1}
I20260929 01:01:38.987193 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:38.976117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:52417)
0929 01:01:38.977059 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:38.977092 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:38.977252 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:38.977643 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:38.977652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:38.978138 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:38.978321 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:38.978332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.980042 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.980048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:38.981067 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:38.981080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.981971 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.981977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:38.982171 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:38.983008 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:38.983027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:38.986886 (+ 3859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":699,"threads_started":1}
W20260929 01:01:38.988106 12420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:38.976292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:52417 (local address 127.3.21.254:34019)
0929 01:01:38.976931 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:38.976937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:38.977208 (+ 271us) server_negotiation.cc:407] Connection header received
0929 01:01:38.977386 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:38.977390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:38.977470 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:38.977553 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:38.978491 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.979863 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.981221 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:38.981812 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:38.981995 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:38.987888 (+ 5893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:52417: BlockingRecv error: recv got EOF from 127.3.21.193:52417 (error 108)
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":411,"spinlock_wait_cycles":11392}
Sep 29 01:01:39 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:39 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:39.491048 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:39.480251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:42995)
0929 01:01:39.481002 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:39.481021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:39.481175 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:39.481884 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:39.481888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:39.482397 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:39.482659 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:39.482675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.484044 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.484050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:39.485176 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:39.485190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.486171 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.486177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:39.486321 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:39.487257 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:39.487278 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:39.490713 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":451,"threads_started":1}
W20260929 01:01:39.491225 12420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:39.480614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:42995 (local address 127.3.21.254:34019)
0929 01:01:39.480844 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:39.480851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:39.481155 (+ 304us) server_negotiation.cc:407] Connection header received
0929 01:01:39.481459 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:39.481467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:39.481589 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:39.481766 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:39.482826 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.483871 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.485326 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.485835 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.486149 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:39.490994 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:42995: BlockingRecv error: recv got EOF from 127.3.21.194:42995 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 29 01:01:40 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:40 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:40 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:40 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:40.010443 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:39.988549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:52943)
0929 01:01:39.989772 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:39.989792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:39.989942 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:39.990685 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:39.990692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:39.991186 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:39.991414 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:39.991425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.992860 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.992866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:39.993934 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:39.993947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.995053 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.995059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:39.995255 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:40.005686 (+ 10431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:40.005717 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:40.009908 (+ 4191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":1002,"threads_started":1}
W20260929 01:01:40.010424 12420 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:39.988566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:52943 (local address 127.3.21.254:34019)
0929 01:01:39.988847 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:39.988859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:39.989919 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 01:01:39.990078 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:39.990084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:39.990158 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:39.990257 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:39.991566 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.992688 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.994107 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.994890 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.995042 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:40.010173 (+ 15131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:52943: BlockingRecv error: recv got EOF from 127.3.21.195:52943 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":82}
I20260929 01:01:40.010615 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:39.988562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:55397)
0929 01:01:39.989540 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:39.989561 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:39.989707 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:39.990155 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:39.990162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:39.990892 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:39.991190 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:39.991205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.992927 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.992933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:39.993972 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:39.993986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.995093 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.995100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:39.995306 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:40.005686 (+ 10380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:40.005717 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:40.010132 (+ 4415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":713,"threads_started":1}
W20260929 01:01:40.010808 12428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:39.988677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:55397 (local address 127.3.21.254:34019)
0929 01:01:39.989357 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:39.989366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:39.989773 (+ 407us) server_negotiation.cc:407] Connection header received
0929 01:01:39.989858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:39.989867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:39.989954 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:39.990074 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:39.991408 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.992753 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.994158 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:39.994930 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:39.995076 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:40.010372 (+ 15296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:55397: BlockingRecv error: recv got EOF from 127.3.21.193:55397 (error 108)
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":527,"spinlock_wait_cycles":7424,"thread_start_us":358,"threads_started":1}
Sep 29 01:01:40 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:40 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:40.502234 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:40.492360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:52317)
0929 01:01:40.493064 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:40.493087 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:40.493247 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:40.493644 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:40.493648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:40.494145 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:40.494315 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:40.494326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:40.495488 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:40.495498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:40.496609 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:40.496624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:40.497440 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:40.497448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:40.497649 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:40.498331 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:40.498350 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:40.501961 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":391,"threads_started":1}
W20260929 01:01:40.502475 12428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:40.492778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:52317 (local address 127.3.21.254:34019)
0929 01:01:40.493066 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:40.493074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:40.493192 (+ 118us) server_negotiation.cc:407] Connection header received
0929 01:01:40.493406 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:40.493415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:40.493494 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:40.493612 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:40.494524 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:40.495333 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:40.496774 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:40.497256 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:40.497406 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:40.502216 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:52317: BlockingRecv error: recv got EOF from 127.3.21.194:52317 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 29 01:01:41 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:41 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:41 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:41 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:41.025606 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:41.011852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:34775)
0929 01:01:41.013303 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:41.013317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:41.013422 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:41.013794 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:41.013809 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:41.014274 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:41.014557 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:41.014571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.016217 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.016223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:41.017762 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:41.017780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.019297 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.019303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:41.019492 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:41.020246 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:41.020273 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:41.025249 (+ 4976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":1251,"threads_started":1}
I20260929 01:01:41.025682 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:41.011852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:36121)
0929 01:01:41.012695 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:41.012722 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:41.012901 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:41.013546 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:41.013553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:41.014099 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:41.014298 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:41.014311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.016217 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.016223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:41.017865 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:41.017885 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.019234 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.019242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:41.019597 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:41.020301 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:41.020326 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:41.025299 (+ 4973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":336,"threads_started":1}
W20260929 01:01:41.026212 12433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:41.012067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:34775 (local address 127.3.21.254:34019)
0929 01:01:41.012730 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:41.012740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:41.013407 (+ 667us) server_negotiation.cc:407] Connection header received
0929 01:01:41.013571 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:41.013581 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:41.013636 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:41.013726 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:41.014698 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.015904 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.018040 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.019131 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.019306 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:41.025699 (+ 6393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:34775: BlockingRecv error: recv got EOF from 127.3.21.195:34775 (error 108)
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":515,"thread_start_us":841,"threads_started":1}
W20260929 01:01:41.026357 12432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:41.012135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:36121 (local address 127.3.21.254:34019)
0929 01:01:41.013178 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:41.013183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:41.013199 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:41.013253 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:41.013258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:41.013340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:41.013488 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:41.014601 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.015905 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.018040 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.019002 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.019153 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:41.025713 (+ 6560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:36121: BlockingRecv error: recv got EOF from 127.3.21.193:36121 (error 108)
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":900,"thread_start_us":341,"threads_started":1}
Sep 29 01:01:41 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:41 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:41.513162 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:41.503532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:33435)
0929 01:01:41.504125 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:41.504142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:41.504258 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:41.504685 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:41.504688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:41.505157 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:41.505318 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:41.505327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.506753 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.506758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:41.507916 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:41.507931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.508748 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.508753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:41.508893 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:41.509617 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:41.509636 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:41.512879 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":324,"threads_started":1}
W20260929 01:01:41.513547 12432 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:41.503584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:33435 (local address 127.3.21.254:34019)
0929 01:01:41.503779 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:41.503783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:41.504237 (+ 454us) server_negotiation.cc:407] Connection header received
0929 01:01:41.504473 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:41.504477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:41.504539 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:41.504624 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:41.505578 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.506538 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.508075 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:41.508575 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:41.508682 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:41.513154 (+ 4472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:33435: BlockingRecv error: recv got EOF from 127.3.21.194:33435 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 01:01:42 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:42 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:42 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:42 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:42.052321 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:42.027458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:48803)
0929 01:01:42.040071 (+ 12613us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:42.040102 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:42.040309 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:42.041301 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:42.041379 (+ 78us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:42.042087 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:42.042296 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:42.042306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.043716 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.043721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:42.044973 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:42.044990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.045951 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.045960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:42.046162 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:42.047131 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:42.047166 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:42.051943 (+ 4777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12485,"thread_start_us":12359,"threads_started":1}
I20260929 01:01:42.052870 12438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:42.027451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:32831)
0929 01:01:42.041025 (+ 13574us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:42.041047 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:42.041956 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:42.042577 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:42.042581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:42.042983 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:42.043162 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:42.043172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.044581 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.044586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:42.045851 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:42.045872 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.047081 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.047092 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:42.047622 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:42.048263 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:42.048290 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:42.052453 (+ 4163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13366,"thread_start_us":13125,"threads_started":1}
W20260929 01:01:42.053259 12435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:42.027571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:48803 (local address 127.3.21.254:34019)
0929 01:01:42.040702 (+ 13131us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:42.040708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:42.040728 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:01:42.040777 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:42.040782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:42.041120 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:42.041246 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:42.042436 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.043369 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.045189 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.045761 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.045878 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:42.052350 (+ 6472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:48803: BlockingRecv error: recv got EOF from 127.3.21.195:48803 (error 108)
Metrics: {"server-negotiator.queue_time_us":12996,"thread_start_us":11987,"threads_started":1}
W20260929 01:01:42.053346 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:42.027494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:32831 (local address 127.3.21.254:34019)
0929 01:01:42.039786 (+ 12292us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:42.039795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:42.041207 (+ 1412us) server_negotiation.cc:407] Connection header received
0929 01:01:42.042243 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:42.042248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:42.042317 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:42.042437 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:42.043318 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.044400 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.046136 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.046769 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.046894 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:42.053069 (+ 6175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:32831: BlockingRecv error: recv got EOF from 127.3.21.193:32831 (error 108)
Metrics: {"server-negotiator.queue_time_us":12123,"thread_start_us":12878,"threads_started":1}
Sep 29 01:01:42 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:42 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:42.524688 12439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:42.514117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:60643)
0929 01:01:42.514823 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:42.514841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:42.514985 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:42.515376 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:42.515380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:42.515896 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:42.516082 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:42.516092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.517234 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.517238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:42.518310 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:42.518323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.519400 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.519405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:42.519553 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:42.520457 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:42.520490 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:42.524165 (+ 3675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":430,"threads_started":1}
W20260929 01:01:42.524722 12435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:42.514238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:60643 (local address 127.3.21.254:34019)
0929 01:01:42.514512 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:42.514522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:42.514950 (+ 428us) server_negotiation.cc:407] Connection header received
0929 01:01:42.515117 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:42.515121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:42.515189 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:42.515280 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:42.516248 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.517024 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.518512 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:42.519139 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:42.519254 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:42.524480 (+ 5226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:60643: BlockingRecv error: recv got EOF from 127.3.21.194:60643 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 01:01:43 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:43 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:43 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:43 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:43.064314 12440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:43.053472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:34501)
0929 01:01:43.054403 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:43.054437 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:43.054708 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:43.055232 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:43.055237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:43.055882 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:43.056121 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:43.056134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.057364 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.057370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:43.058298 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:43.058310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.059070 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.059076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:43.059251 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:43.059886 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:43.059904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:43.063921 (+ 4017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":497,"threads_started":1}
W20260929 01:01:43.064868 12441 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:43.053525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:34501 (local address 127.3.21.254:34019)
0929 01:01:43.054934 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:43.054939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:43.054955 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:43.055002 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:43.055006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:43.055081 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:43.055202 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:43.056274 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.057225 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.058454 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.058911 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.059050 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:43.064603 (+ 5553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:34501: BlockingRecv error: recv got EOF from 127.3.21.195:34501 (error 108)
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":609,"threads_started":1}
I20260929 01:01:43.065230 12442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:43.054635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:41429)
0929 01:01:43.055872 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:43.055893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:43.056009 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:43.056475 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:43.056479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:43.056949 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:43.057112 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:43.057121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.058192 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.058198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:43.059287 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:43.059299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.060070 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.060078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:43.060244 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:43.060858 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:43.060877 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:43.064913 (+ 4036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":944,"threads_started":1}
W20260929 01:01:43.066207 12443 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:43.054725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:41429 (local address 127.3.21.254:34019)
0929 01:01:43.056205 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:43.056210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:43.056224 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:01:43.056263 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:43.056266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:43.056315 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:43.056447 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:43.057264 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.058039 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.059434 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.059883 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.060006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:43.065916 (+ 5910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:41429: BlockingRecv error: recv got EOF from 127.3.21.193:41429 (error 108)
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":1270,"threads_started":1}
Sep 29 01:01:43 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:43 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:43.535388 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:43.525907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:40867)
0929 01:01:43.526983 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:43.527000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:43.527183 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:43.527706 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:43.527710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:43.528208 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:43.528360 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:43.528369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.529562 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.529568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:43.530469 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:43.530480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.531256 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.531263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:43.531415 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:43.532129 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:43.532147 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:43.535146 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":720,"threads_started":1}
W20260929 01:01:43.535627 12443 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:43.526114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:40867 (local address 127.3.21.254:34019)
0929 01:01:43.526337 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:43.526341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:43.527182 (+ 841us) server_negotiation.cc:407] Connection header received
0929 01:01:43.527339 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:43.527343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:43.527405 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:43.527520 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:43.528510 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.529407 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.530657 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:43.531057 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:43.531192 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:43.535359 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:40867: BlockingRecv error: recv got EOF from 127.3.21.194:40867 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 01:01:44 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:44 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:44 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:44.076133 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:44.065253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:37419)
0929 01:01:44.065925 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:44.065946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:44.066080 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:44.066823 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:44.066828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:44.067321 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:44.067512 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:44.067526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.069000 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.069006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:44.070603 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:44.070616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.071456 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.071462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:44.071641 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:44.072304 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:44.072327 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:44.075789 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":256,"threads_started":1}
Sep 29 01:01:44 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:01:44.076776 12446 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:44.065404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:37419 (local address 127.3.21.254:34019)
0929 01:01:44.066086 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:44.066092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:44.066112 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:01:44.066472 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:44.066496 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:44.066602 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:44.066834 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:44.067693 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.068817 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.070784 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.071264 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.071405 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:44.076457 (+ 5052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:37419: BlockingRecv error: recv got EOF from 127.3.21.195:37419 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":436,"threads_started":1}
I20260929 01:01:44.077736 12448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:44.066300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:45549)
0929 01:01:44.068181 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:44.068211 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:44.068343 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:44.069301 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:44.069308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:44.069855 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:44.070053 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:44.070065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.071246 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.071252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:44.072236 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:44.072248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.073009 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.073015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:44.073216 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:44.073819 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:44.073844 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:44.077382 (+ 3538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1735,"thread_start_us":1586,"threads_started":1}
W20260929 01:01:44.077862 12447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:44.066234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:45549 (local address 127.3.21.254:34019)
0929 01:01:44.067601 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:44.067608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:44.068293 (+ 685us) server_negotiation.cc:407] Connection header received
0929 01:01:44.068985 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:44.068989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:44.069065 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:44.069167 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:44.070240 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.071090 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.072390 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.072851 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.072984 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:44.077602 (+ 4618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:45549: BlockingRecv error: recv got EOF from 127.3.21.193:45549 (error 108)
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":360,"threads_started":1}
Sep 29 01:01:44 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:44 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:44.545742 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:44.536474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:49403)
0929 01:01:44.537059 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:44.537076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:44.537233 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:44.537706 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:44.537711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:44.538169 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:44.538336 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:44.538345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.539529 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.539536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:44.540615 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:44.540627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.541540 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.541546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:44.541770 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:44.542463 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:44.542501 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:44.545472 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":303,"threads_started":1}
W20260929 01:01:44.546057 12447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:44.536565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:49403 (local address 127.3.21.254:34019)
0929 01:01:44.536766 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:44.536774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:44.537168 (+ 394us) server_negotiation.cc:407] Connection header received
0929 01:01:44.537380 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:44.537385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:44.537456 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:44.537661 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:44.538501 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.539389 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.540754 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:44.541346 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:44.541490 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:44.545779 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:49403: BlockingRecv error: recv got EOF from 127.3.21.194:49403 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 01:01:45 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:45 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:45 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:45 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:45.099341 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:45.078927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:40495)
0929 01:01:45.079544 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:45.079570 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:45.079821 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:45.084159 (+ 4338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:45.084168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:45.084640 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:45.084784 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:45.084794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.086047 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.086052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:45.087492 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:45.087506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.088454 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.088463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:45.088753 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:45.094367 (+ 5614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:45.094398 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:45.098964 (+ 4566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
I20260929 01:01:45.099433 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:45.077349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:37913)
0929 01:01:45.078288 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:45.078310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:45.078432 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:45.083359 (+ 4927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:45.083365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:45.083891 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:45.084115 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:45.084128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.085571 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.085577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:45.087103 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:45.087120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.088010 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.088015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:45.088201 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:45.094368 (+ 6167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:45.094398 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:45.099064 (+ 4666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":446,"threads_started":1}
W20260929 01:01:45.099524 12452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:45.078951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:40495 (local address 127.3.21.254:34019)
0929 01:01:45.083859 (+ 4908us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:45.083863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:45.083878 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:45.083915 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:45.083919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:45.083968 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:45.084054 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:45.084932 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.085893 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.087657 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.088285 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.088378 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:45.099189 (+ 10811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:40495: BlockingRecv error: recv got EOF from 127.3.21.193:40495 (error 108)
Metrics: {"server-negotiator.queue_time_us":4812,"thread_start_us":4668,"threads_started":1}
W20260929 01:01:45.100008 12451 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:45.077538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:37913 (local address 127.3.21.254:34019)
0929 01:01:45.082961 (+ 5423us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:45.082967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:45.082984 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:01:45.083033 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:45.083038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:45.083156 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:45.083312 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:45.084297 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.085402 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.087286 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.087841 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.087973 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:45.099378 (+ 11405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:37913: BlockingRecv error: recv got EOF from 127.3.21.195:37913 (error 108)
Metrics: {"server-negotiator.queue_time_us":5291,"thread_start_us":5189,"threads_started":1}
Sep 29 01:01:45 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:45 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:45.556459 12454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:45.546819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:35225)
0929 01:01:45.547585 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:45.547607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:45.547798 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:45.548197 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:45.548202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:45.548601 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:45.548746 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:45.548756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.549915 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.549928 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:45.551082 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:45.551095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.551818 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.551825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:45.551996 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:45.552667 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:45.552686 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:45.556081 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":529,"threads_started":1}
W20260929 01:01:45.556541 12451 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:45.547098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:35225 (local address 127.3.21.254:34019)
0929 01:01:45.547416 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:45.547424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:45.547717 (+ 293us) server_negotiation.cc:407] Connection header received
0929 01:01:45.547951 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:45.547960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:45.548024 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:45.548141 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:45.548906 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.549774 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.551231 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:45.551654 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:45.551788 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:45.556296 (+ 4508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:35225: BlockingRecv error: recv got EOF from 127.3.21.194:35225 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 01:01:46 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:46 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:46 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:46 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:46.119369 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:46.100567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:35019)
0929 01:01:46.108041 (+ 7474us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:46.108064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:46.108213 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:46.108989 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:46.108994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:46.109486 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:46.109708 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:46.109720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.111337 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.111341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:46.113001 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:46.113018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.114063 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.114070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:46.114209 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:46.115039 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:46.115069 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:46.118973 (+ 3904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7352,"thread_start_us":7129,"threads_started":1}
I20260929 01:01:46.119774 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:46.100567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:49275)
0929 01:01:46.107796 (+ 7229us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:46.107829 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:46.108018 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:46.109664 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:46.109669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:46.110180 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:46.110352 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:46.110361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.111885 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.111891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:46.113357 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:46.113370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.114511 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.114522 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:46.114744 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:46.115400 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:46.115429 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:46.119396 (+ 3967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7015,"thread_start_us":6835,"threads_started":1}
W20260929 01:01:46.120038 12457 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:46.100734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:49275 (local address 127.3.21.254:34019)
0929 01:01:46.109087 (+ 8353us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:46.109095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:46.109114 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:01:46.109180 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:46.109184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:46.109261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:46.109386 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:46.110530 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.111728 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.113501 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.114299 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.114389 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:46.119780 (+ 5391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:49275: BlockingRecv error: recv got EOF from 127.3.21.193:49275 (error 108)
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":8240,"thread_start_us":7377,"threads_started":1}
W20260929 01:01:46.120122 12458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:46.100718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:35019 (local address 127.3.21.254:34019)
0929 01:01:46.108399 (+ 7681us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:46.108404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:46.108419 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:46.108479 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:46.108483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:46.108539 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:46.108673 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:46.109917 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.111114 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.113170 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.113859 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.114003 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:46.119222 (+ 5219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:35019: BlockingRecv error: recv got EOF from 127.3.21.195:35019 (error 108)
Metrics: {"server-negotiator.queue_time_us":7518,"thread_start_us":8083,"threads_started":1}
Sep 29 01:01:46 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:46 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:46.567458 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:46.557499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:36229)
0929 01:01:46.558297 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:46.558316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:46.558445 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:46.558995 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:46.558999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:46.559446 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:46.559620 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:46.559629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.560866 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.560871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:46.561840 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:46.561854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.562871 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.562876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:46.563021 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:46.563796 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:46.563814 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:46.567089 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":569,"threads_started":1}
W20260929 01:01:46.567543 12458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:46.557554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:36229 (local address 127.3.21.254:34019)
0929 01:01:46.557777 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:46.557784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:46.558413 (+ 629us) server_negotiation.cc:407] Connection header received
0929 01:01:46.558734 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:46.558738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:46.558814 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:46.558913 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:46.559816 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.560612 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.562042 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:46.562540 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:46.562798 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:46.567294 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:36229: BlockingRecv error: recv got EOF from 127.3.21.194:36229 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 01:01:47 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:47 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:47 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:47 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:47.132571 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:47.120687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:46311)
0929 01:01:47.122813 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:47.122838 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:47.122977 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:47.123420 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:47.123428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:47.123998 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:47.124224 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:47.124260 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.125636 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.125642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:47.126700 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:47.126714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.127559 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.127566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:47.127777 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:47.128477 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:47.128505 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:47.132094 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1957,"thread_start_us":1798,"threads_started":1}
I20260929 01:01:47.132859 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:47.121010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:51539)
0929 01:01:47.122463 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:47.122514 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:47.122689 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:47.123509 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:47.123516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:47.123998 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:47.124214 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:47.124227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.125685 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.125691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:47.126719 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:47.126734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.127647 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.127655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:47.127907 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:47.128682 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:47.128711 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:47.132409 (+ 3698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1274,"thread_start_us":1112,"threads_started":1}
W20260929 01:01:47.133001 12462 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:47.120733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:46311 (local address 127.3.21.254:34019)
0929 01:01:47.121869 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:47.121880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:47.122945 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 01:01:47.123112 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:47.123118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:47.123209 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:47.123336 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:47.124420 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.125477 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.126852 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.127378 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.127522 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:47.132426 (+ 4904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:46311: BlockingRecv error: recv got EOF from 127.3.21.195:46311 (error 108)
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":945,"thread_start_us":2093,"threads_started":1}
W20260929 01:01:47.133673 12461 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:47.121011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:51539 (local address 127.3.21.254:34019)
0929 01:01:47.123178 (+ 2167us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:47.123184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:47.123199 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:01:47.123247 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:47.123254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:47.123337 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:47.123437 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:47.124387 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.125509 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.126890 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.127464 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.127619 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:47.133438 (+ 5819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:51539: BlockingRecv error: recv got EOF from 127.3.21.193:51539 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2008,"spinlock_wait_cycles":70400,"thread_start_us":494,"threads_started":1}
Sep 29 01:01:47 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:47 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:47.578387 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:47.568692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:54841)
0929 01:01:47.569516 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:47.569533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:47.569714 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:47.570207 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:47.570214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:47.570675 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:47.570840 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:47.570850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.571943 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.571950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:47.572880 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:47.572892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.573665 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.573671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:47.573956 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:47.574887 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:47.574938 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:47.578126 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":564,"threads_started":1}
W20260929 01:01:47.578856 12461 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:47.568988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:54841 (local address 127.3.21.254:34019)
0929 01:01:47.569343 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:47.569352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:47.569662 (+ 310us) server_negotiation.cc:407] Connection header received
0929 01:01:47.569975 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:47.569979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:47.570042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:47.570148 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:47.570998 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.571794 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.573034 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:47.573497 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:47.573638 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:47.578395 (+ 4757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:54841: BlockingRecv error: recv got EOF from 127.3.21.194:54841 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Sep 29 01:01:48 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:48 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:48 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:48 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:48.159422 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:48.133911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:60013)
0929 01:01:48.149689 (+ 15778us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:48.149713 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:48.149878 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:48.150619 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:48.150625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:48.151183 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:48.151431 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:48.151447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.152702 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.152709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:48.153752 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:48.153764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.154557 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.154563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:48.154807 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:48.155531 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:48.155551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:48.159128 (+ 3577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15642,"thread_start_us":15937,"threads_started":1}
W20260929 01:01:48.159791 12468 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:48.134010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:60013 (local address 127.3.21.254:34019)
0929 01:01:48.149459 (+ 15449us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:48.149465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:48.150229 (+ 764us) server_negotiation.cc:407] Connection header received
0929 01:01:48.150295 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:48.150300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:48.150391 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:48.150554 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:48.151605 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.152539 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.153917 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.154366 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.154548 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:48.159450 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:60013: BlockingRecv error: recv got EOF from 127.3.21.195:60013 (error 108)
Metrics: {"server-negotiator.queue_time_us":15301,"thread_start_us":16831,"threads_started":1}
I20260929 01:01:48.159905 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:48.134028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:51243)
0929 01:01:48.149893 (+ 15865us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:48.149914 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:48.150106 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:48.151430 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:48.151438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:48.151873 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:48.152044 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:48.152053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.153348 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.153354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:48.154294 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:48.154307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.155167 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.155175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:48.155395 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:48.156056 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:48.156073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:48.159514 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15736,"thread_start_us":15589,"threads_started":1}
W20260929 01:01:48.160285 12466 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:48.134057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:51243 (local address 127.3.21.254:34019)
0929 01:01:48.151121 (+ 17064us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:48.151128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:48.151144 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:48.151184 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:48.151188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:48.151267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:48.151393 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:48.152198 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.153206 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.154443 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.154965 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.155093 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:48.159936 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:51243: BlockingRecv error: recv got EOF from 127.3.21.193:51243 (error 108)
Metrics: {"server-negotiator.queue_time_us":16892,"thread_start_us":15138,"threads_started":1}
Sep 29 01:01:48 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:48 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:48.589246 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:48.579465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:42317)
0929 01:01:48.580276 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:48.580292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:48.580465 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:48.580858 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:48.580863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:48.581282 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:48.581438 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:48.581446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.582539 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.582544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:48.583505 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:48.583516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.584255 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.584261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:48.584432 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:48.585287 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:48.585309 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:48.588994 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":526,"threads_started":1}
W20260929 01:01:48.589430 12466 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:48.579682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:42317 (local address 127.3.21.254:34019)
0929 01:01:48.580006 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:48.580014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:48.580397 (+ 383us) server_negotiation.cc:407] Connection header received
0929 01:01:48.580611 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:48.580616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:48.580684 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:48.580815 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:48.581591 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.582353 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.583651 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:48.584061 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:48.584194 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:48.589196 (+ 5002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:42317: BlockingRecv error: recv got EOF from 127.3.21.194:42317 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Sep 29 01:01:49 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:49 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:49 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:49 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:49.174350 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:49.161022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:58549)
0929 01:01:49.162625 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:49.162642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:49.162828 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:49.163298 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:49.163305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:49.163783 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:49.164009 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:49.164023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.165638 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.165644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:49.167297 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:49.167315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.168801 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.168808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:49.169039 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:49.169847 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:49.169874 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:49.173981 (+ 4107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1438,"thread_start_us":1281,"threads_started":1}
I20260929 01:01:49.174669 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:49.161016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:45763)
0929 01:01:49.161834 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:49.161860 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:49.162035 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:49.163067 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:49.163075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:49.163469 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:49.163756 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:49.163770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.165439 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.165448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:49.167213 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:49.167231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.168752 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.168760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:49.168968 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:49.169822 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:49.169853 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:49.174259 (+ 4406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":321,"threads_started":1}
W20260929 01:01:49.174705 12472 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:49.161147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:58549 (local address 127.3.21.254:34019)
0929 01:01:49.162127 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:49.162137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:49.162767 (+ 630us) server_negotiation.cc:407] Connection header received
0929 01:01:49.162996 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:49.163004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:49.163123 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:49.163257 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:49.164183 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.165459 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.167435 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.168544 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.168753 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:49.174290 (+ 5537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:58549: BlockingRecv error: recv got EOF from 127.3.21.195:58549 (error 108)
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":804,"spinlock_wait_cycles":2816,"thread_start_us":591,"threads_started":1}
W20260929 01:01:49.175009 12473 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:49.162006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:45763 (local address 127.3.21.254:34019)
0929 01:01:49.162698 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:49.162703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:49.162719 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:01:49.162774 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:49.162803 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:49.162868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:49.163013 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:49.163952 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.165273 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.167408 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.168547 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.168750 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:49.174550 (+ 5800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:45763: BlockingRecv error: recv got EOF from 127.3.21.193:45763 (error 108)
Metrics: {"mutex_wait_us":391,"server-negotiator.queue_time_us":605,"thread_start_us":388,"threads_started":1}
Sep 29 01:01:49 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:49 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:49.599985 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:49.590516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:58135)
0929 01:01:49.591261 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:49.591278 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:49.591422 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:49.591774 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:49.591779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:49.592167 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:49.592342 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:49.592351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.593411 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.593420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:49.594342 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:49.594354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.595093 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.595099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:49.595302 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:49.596081 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:49.596103 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:49.599705 (+ 3602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":473,"threads_started":1}
W20260929 01:01:49.600463 12473 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:49.590599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:58135 (local address 127.3.21.254:34019)
0929 01:01:49.590809 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:49.590815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:49.591362 (+ 547us) server_negotiation.cc:407] Connection header received
0929 01:01:49.591560 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:49.591565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:49.591637 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:49.591741 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:49.592495 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.593263 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.594527 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:49.594931 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:49.595075 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:49.600270 (+ 5195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:58135: BlockingRecv error: recv got EOF from 127.3.21.194:58135 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 01:01:50 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:50 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:50 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.195@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:50 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.193@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:50.193697 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:50.175678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.195:45665)
0929 01:01:50.176521 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:50.176543 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:50.176777 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:50.177353 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:50.177388 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:50.177900 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:50.178117 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:50.178128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.179608 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.179614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:50.180828 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:50.180841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.181770 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.181777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:50.182107 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.189744 (+ 7637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:50.189769 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:50.193363 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":566,"threads_started":1}
W20260929 01:01:50.193950 12476 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:50.175789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:45665 (local address 127.3.21.254:34019)
0929 01:01:50.176349 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:50.176355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:50.176634 (+ 279us) server_negotiation.cc:407] Connection header received
0929 01:01:50.177054 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:50.177061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:50.177154 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:50.177267 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:50.178281 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.179466 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.180974 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.181622 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.181723 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.193573 (+ 11850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.195:45665: BlockingRecv error: recv got EOF from 127.3.21.195:45665 (error 108)
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":426,"thread_start_us":262,"threads_started":1}
I20260929 01:01:50.194024 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:50.176167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.193:35619)
0929 01:01:50.177295 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:50.177312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:50.177440 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:50.177790 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:50.177794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:50.178217 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:50.178573 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:50.178587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.179977 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.179983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:50.181188 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:50.181202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.182073 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.182077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:50.182197 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.189752 (+ 7555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:50.189777 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:50.193686 (+ 3909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":879,"threads_started":1}
W20260929 01:01:50.194362 12478 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:50.176499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:35619 (local address 127.3.21.254:34019)
0929 01:01:50.177103 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:50.177109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:50.177396 (+ 287us) server_negotiation.cc:407] Connection header received
0929 01:01:50.177553 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:50.177559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:50.177631 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:50.177747 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:50.178731 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.179845 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.181324 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.181910 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.182032 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.194049 (+ 12017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.193:35619: BlockingRecv error: recv got EOF from 127.3.21.193:35619 (error 108)
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":444,"thread_start_us":276,"threads_started":1}
Sep 29 01:01:50 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:01:50 dist-test-slave-k0b7 krb5kdc[11407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.21.194@KRBTEST.COM for oryx/127.3.21.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:01:50.611240 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:50.601370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34019 (local address 127.3.21.194:51507)
0929 01:01:50.602066 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:50.602082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:50.602255 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:50.602719 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:50.602723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:50.603322 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:50.603563 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:50.603576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.604947 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.604956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:50.605939 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:50.605951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.606811 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.606815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:50.606997 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.607718 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:50.607737 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:50.610963 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.21.254:34019: Server oryx/127.3.21.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":474,"threads_started":1}
W20260929 01:01:50.611416 12478 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:50.601568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:51507 (local address 127.3.21.254:34019)
0929 01:01:50.601798 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:50.601804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:50.602189 (+ 385us) server_negotiation.cc:407] Connection header received
0929 01:01:50.602425 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:50.602430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:50.602555 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:50.602659 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:50.603747 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.604711 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.606137 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.606668 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.606769 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.611185 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.21.194:51507: BlockingRecv error: recv got EOF from 127.3.21.194:51507 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:01:50.923499 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11996
I20260929 01:01:50.937427 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:50.924708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.3.21.193:32813)
0929 01:01:50.925493 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:50.925500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:50.925528 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:01:50.925588 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:50.925595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:50.925829 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:50.925990 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:50.927529 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.928702 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.930034 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.930885 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.931010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.934012 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:50.934024 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:50.934039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:50.934078 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:50.936893 (+ 2815us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.21.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 01:01:50.937192 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34362: Request ticket server kudu/127.3.21.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":440,"threads_started":1}
W20260929 01:01:50.937569 12481 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34362: Request ticket server kudu/127.3.21.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 01:01:50.937775 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11996: Not authorized: Client connection negotiation failed: client connection to 127.3.21.193:32813: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.21.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 01:01:50.938076 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11996
I20260929 01:01:50.955736 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12147
I20260929 01:01:50.969449 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:50.956944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.3.21.194:44295)
0929 01:01:50.957548 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:50.957557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:50.957586 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:01:50.957644 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:50.957651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:50.957873 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:50.958087 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:50.959594 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.960800 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.962236 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.963095 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.963213 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.965996 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:50.966007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:50.966023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:50.966068 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:50.968815 (+ 2747us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.21.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 01:01:50.969160 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49636: Request ticket server kudu/127.3.21.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":301,"threads_started":1}
W20260929 01:01:50.969563 12482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49636: Request ticket server kudu/127.3.21.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 01:01:50.969656 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12147: Not authorized: Client connection negotiation failed: client connection to 127.3.21.194:44295: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.21.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 01:01:50.969851 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12147
I20260929 01:01:50.986843 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12278
I20260929 01:01:51.000089 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:50.988064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.3.21.195:36163)
0929 01:01:50.988641 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:50.988649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:50.988675 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:01:50.988736 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:50.988743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:50.989080 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:50.989258 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:50.990688 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.991887 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.993134 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:50.993723 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:50.993830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:50.996704 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:50.996713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:50.996729 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:50.996774 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:50.999544 (+ 2770us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.21.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 01:01:50.999825 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52672: Request ticket server kudu/127.3.21.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":288,"threads_started":1}
W20260929 01:01:51.000238 12483 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52672: Request ticket server kudu/127.3.21.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 01:01:51.000494 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12278: Not authorized: Client connection negotiation failed: client connection to 127.3.21.195:36163: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.21.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 01:01:51.000705 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12278
I20260929 01:01:51.018684 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 11927
I20260929 01:01:51.020097 11989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:51.141667 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 11927
2026-09-29T01:01:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21687 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T01:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:51Z Disabled control of system clock
I20260929 01:01:51.227115 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:46267
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 01:01:51.584542 12488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:51.584875 12488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:51.585031 12488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:51.593071 12488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:51.593216 12488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:51.593393 12488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:51.593495 12488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:51.593614 12488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:51.603590 12488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:46267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:51.605265 12488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:51.607707 12488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:51.622447 12494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:51.622462 12496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:51.622418 12493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:51.623489 12488 server_base.cc:1034] running on GCE node
I20260929 01:01:51.624583 12488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:51.626300 12488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:51.627570 12488 hybrid_clock.cc:648] HybridClock initialized: now 1790643711627508 us; error 108 us; skew 500 ppm
I20260929 01:01:51.627990 12488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:51.630762 12488 webserver.cc:460] Webserver started at http://127.3.21.254:40947/ using document root <none> and password file <none>
I20260929 01:01:51.631582 12488 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:51.631731 12488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:51.632107 12488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:51.635560 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "ed5ce471863c45f786be35e79f3bbc9a"
format_stamp: "Formatted at 2026-09-29 01:01:51 on dist-test-slave-k0b7"
server_key: "b0be3bc91b9b95643d4d80fec0453669"
server_key_iv: "d8cd73bfc3c5e2a17a79d4bcbeee4390"
server_key_version: "encryptionkey@0"
I20260929 01:01:51.636477 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "ed5ce471863c45f786be35e79f3bbc9a"
format_stamp: "Formatted at 2026-09-29 01:01:51 on dist-test-slave-k0b7"
server_key: "b0be3bc91b9b95643d4d80fec0453669"
server_key_iv: "d8cd73bfc3c5e2a17a79d4bcbeee4390"
server_key_version: "encryptionkey@0"
I20260929 01:01:51.642562 12488 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:01:51.647106 12502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:51.648931 12488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:51.649173 12488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "ed5ce471863c45f786be35e79f3bbc9a"
format_stamp: "Formatted at 2026-09-29 01:01:51 on dist-test-slave-k0b7"
server_key: "b0be3bc91b9b95643d4d80fec0453669"
server_key_iv: "d8cd73bfc3c5e2a17a79d4bcbeee4390"
server_key_version: "encryptionkey@0"
I20260929 01:01:51.649434 12488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:51.707719 12488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:51.709050 12488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:51.709394 12488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:51.730032 12488 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:46267
I20260929 01:01:51.730086 12553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:46267 every 8 connection(s)
I20260929 01:01:51.731602 12488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:51.733806 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12488
I20260929 01:01:51.734112 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:51.734817 3159 external_mini_cluster.cc:1428] Setting key 9a9411e331b1bf4e1767aad4ea6f1c43
I20260929 01:01:51.736147 12554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:51.749653 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:51.737058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.3.21.254:46267)
0929 01:01:51.738252 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:51.738269 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:51.738405 (+ 136us) server_negotiation.cc:407] Connection header received
0929 01:01:51.740093 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:51.740114 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:51.740644 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:51.741104 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:51.741693 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:51.743227 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:51.744690 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:51.745422 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:51.745660 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:51.747022 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:51.747030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:51.747050 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:51.747717 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:51.747809 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:51.747833 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:51.747988 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:51.748242 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 01:01:51.748611 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":420,"threads_started":1}
I20260929 01:01:51.750576 12554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a: Bootstrap starting.
I20260929 01:01:51.754321 12554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:51.755504 12554 log.cc:824] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:51.758874 12554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a: No bootstrap required, opened a new log
I20260929 01:01:51.767505 12554 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ce471863c45f786be35e79f3bbc9a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46267 } }
I20260929 01:01:51.767939 12554 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:51.768082 12554 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5ce471863c45f786be35e79f3bbc9a, State: Initialized, Role: FOLLOWER
I20260929 01:01:51.768759 12554 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ce471863c45f786be35e79f3bbc9a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46267 } }
I20260929 01:01:51.769026 12554 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:51.769186 12554 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:51.769402 12554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:51.772199 12554 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ce471863c45f786be35e79f3bbc9a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46267 } }
I20260929 01:01:51.772832 12554 leader_election.cc:302] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5ce471863c45f786be35e79f3bbc9a; no voters:
I20260929 01:01:51.773603 12554 leader_election.cc:288] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:51.773705 12559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:51.774714 12559 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [term 1 LEADER]: Becoming Leader. State: Replica: ed5ce471863c45f786be35e79f3bbc9a, State: Running, Role: LEADER
I20260929 01:01:51.775309 12559 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ce471863c45f786be35e79f3bbc9a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46267 } }
I20260929 01:01:51.776192 12554 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:51.778460 12560 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5ce471863c45f786be35e79f3bbc9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ce471863c45f786be35e79f3bbc9a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46267 } } }
I20260929 01:01:51.778981 12560 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:51.778455 12561 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5ce471863c45f786be35e79f3bbc9a. Latest consensus state: current_term: 1 leader_uuid: "ed5ce471863c45f786be35e79f3bbc9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ce471863c45f786be35e79f3bbc9a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46267 } } }
I20260929 01:01:51.779109 12561 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:51.785002 12566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:51.791857 12566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:51.804149 12566 catalog_manager.cc:1347] Generated new cluster ID: c255d82899da4df4ab52da02b365a10b
I20260929 01:01:51.804322 12566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:51.812707 12566 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:51.814409 12566 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:51.822846 12566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5ce471863c45f786be35e79f3bbc9a: Generated new TSK 0
I20260929 01:01:51.824083 12566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:51.842715 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46267
--builtin_ntp_servers=127.3.21.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 01:01:52.206300 12578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:52.206746 12578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:52.206907 12578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:52.214840 12578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:52.214998 12578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:52.215181 12578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:52.224967 12578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:52.226394 12578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:52.228332 12578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:52.243309 12586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:52.243284 12584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:52.243553 12583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:52.244396 12578 server_base.cc:1034] running on GCE node
I20260929 01:01:52.245127 12578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:52.246217 12578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:52.247475 12578 hybrid_clock.cc:648] HybridClock initialized: now 1790643712247396 us; error 71 us; skew 500 ppm
I20260929 01:01:52.247816 12578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:52.251425 12578 webserver.cc:460] Webserver started at http://127.3.21.193:37271/ using document root <none> and password file <none>
I20260929 01:01:52.252235 12578 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:52.252378 12578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:52.252720 12578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:52.256140 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "0d6d8e0ec3aa4315abe73149e1218261"
format_stamp: "Formatted at 2026-09-29 01:01:52 on dist-test-slave-k0b7"
server_key: "82769e0f7fcb261ef8716e04f9655718"
server_key_iv: "ee26e120dfd055cdf575aaf35bb6d4b6"
server_key_version: "encryptionkey@0"
I20260929 01:01:52.256995 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6d8e0ec3aa4315abe73149e1218261"
format_stamp: "Formatted at 2026-09-29 01:01:52 on dist-test-slave-k0b7"
server_key: "82769e0f7fcb261ef8716e04f9655718"
server_key_iv: "ee26e120dfd055cdf575aaf35bb6d4b6"
server_key_version: "encryptionkey@0"
I20260929 01:01:52.262918 12578 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:01:52.267187 12592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:52.268960 12578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:52.269294 12578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "0d6d8e0ec3aa4315abe73149e1218261"
format_stamp: "Formatted at 2026-09-29 01:01:52 on dist-test-slave-k0b7"
server_key: "82769e0f7fcb261ef8716e04f9655718"
server_key_iv: "ee26e120dfd055cdf575aaf35bb6d4b6"
server_key_version: "encryptionkey@0"
I20260929 01:01:52.269556 12578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:52.289705 12578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:52.290782 12578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:52.291126 12578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:52.292199 12578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:52.294003 12578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:52.294142 12578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:52.294276 12578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:52.294366 12578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:52.334137 12578 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:33635
I20260929 01:01:52.334199 12704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:33635 every 8 connection(s)
I20260929 01:01:52.335675 12578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:52.343005 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12578
I20260929 01:01:52.343211 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:52.343941 3159 external_mini_cluster.cc:1428] Setting key a85cb42555e10c34d25b442ed34f7d32
I20260929 01:01:52.347265 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46267
--builtin_ntp_servers=127.3.21.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 01:01:52.351454 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:52.338599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:33865 (local address 127.3.21.254:46267)
0929 01:01:52.339060 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:52.339065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:52.340297 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 01:01:52.342290 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:52.342296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:52.342408 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:52.342565 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:52.343959 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.344844 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.347398 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.348237 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.348424 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:52.350159 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:52.350169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:52.350177 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:52.350207 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:52.350265 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:52.350285 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:52.350391 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:52.350812 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 01:01:52.350946 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":245,"threads_started":1}
I20260929 01:01:52.352519 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:52.339100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:46267 (local address 127.3.21.193:33865)
0929 01:01:52.340115 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:52.340162 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:52.341972 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:52.342888 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:52.342898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:52.343004 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:52.343769 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:52.343788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.345057 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.345063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:52.347190 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:52.347212 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.348512 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.348526 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:52.348758 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:52.349764 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:01:52.349767 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:52.349815 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:01:52.349916 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:52.350427 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:52.350437 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:52.350445 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:01:52.350723 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 01:01:52.351258 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
I20260929 01:01:52.354462 12705 heartbeater.cc:347] Connected to a master server at 127.3.21.254:46267
I20260929 01:01:52.354856 12705 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:52.355769 12705 heartbeater.cc:511] Master 127.3.21.254:46267 requested a full tablet report, sending...
I20260929 01:01:52.358353 12519 ts_manager.cc:194] Registered new tserver with Master: 0d6d8e0ec3aa4315abe73149e1218261 (127.3.21.193:33635)
I20260929 01:01:52.360209 12519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.193:33865
W20260929 01:01:52.704365 12709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:52.704739 12709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:52.704911 12709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:52.712970 12709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:52.713142 12709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:52.713351 12709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:52.722760 12709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:52.724160 12709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:52.725981 12709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:52.741462 12715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:52.741524 12714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:52.742449 12717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:52.748752 12709 server_base.cc:1034] running on GCE node
I20260929 01:01:52.749578 12709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:52.750757 12709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:52.752013 12709 hybrid_clock.cc:648] HybridClock initialized: now 1790643712751910 us; error 72 us; skew 500 ppm
I20260929 01:01:52.752370 12709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:52.755291 12709 webserver.cc:460] Webserver started at http://127.3.21.194:35787/ using document root <none> and password file <none>
I20260929 01:01:52.756062 12709 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:52.756158 12709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:52.756534 12709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:52.759742 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "f79f0297cb7a4dddbc6ce5c6869bdb65"
format_stamp: "Formatted at 2026-09-29 01:01:52 on dist-test-slave-k0b7"
server_key: "43a6aee1530e8bb0cf62a37b0f5c9b0f"
server_key_iv: "bc88ed4bf7255638f295aa162ba5dc42"
server_key_version: "encryptionkey@0"
I20260929 01:01:52.760622 12709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "f79f0297cb7a4dddbc6ce5c6869bdb65"
format_stamp: "Formatted at 2026-09-29 01:01:52 on dist-test-slave-k0b7"
server_key: "43a6aee1530e8bb0cf62a37b0f5c9b0f"
server_key_iv: "bc88ed4bf7255638f295aa162ba5dc42"
server_key_version: "encryptionkey@0"
I20260929 01:01:52.767899 12709 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260929 01:01:52.772387 12723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:52.774314 12709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 01:01:52.774606 12709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "f79f0297cb7a4dddbc6ce5c6869bdb65"
format_stamp: "Formatted at 2026-09-29 01:01:52 on dist-test-slave-k0b7"
server_key: "43a6aee1530e8bb0cf62a37b0f5c9b0f"
server_key_iv: "bc88ed4bf7255638f295aa162ba5dc42"
server_key_version: "encryptionkey@0"
I20260929 01:01:52.774874 12709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:52.803774 12709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:52.804754 12709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:52.805143 12709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:52.806231 12709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:52.807848 12709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:52.807971 12709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:52.808103 12709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:52.808192 12709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:52.845077 12709 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:42195
I20260929 01:01:52.845147 12835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:42195 every 8 connection(s)
I20260929 01:01:52.846660 12709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:52.856437 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12709
I20260929 01:01:52.856635 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:52.857142 3159 external_mini_cluster.cc:1428] Setting key 698c84cb7924a19ae54889512576b125
I20260929 01:01:52.860311 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46267
--builtin_ntp_servers=127.3.21.212:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 01:01:52.861399 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:52.849800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:56481 (local address 127.3.21.254:46267)
0929 01:01:52.850000 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:52.850007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:52.851708 (+ 1701us) server_negotiation.cc:407] Connection header received
0929 01:01:52.853707 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:52.853713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:52.853838 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:52.853961 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:52.855354 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.856116 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.858020 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.858649 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.858779 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:52.860575 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:52.860580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:52.860583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:52.860602 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:52.860632 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:52.860645 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:52.860743 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:52.860986 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 01:01:52.861087 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:01:52.862773 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:52.850404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:46267 (local address 127.3.21.194:56481)
0929 01:01:52.851532 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:52.851571 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:52.853347 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:52.854259 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:52.854269 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:52.854404 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:52.855180 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:52.855199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.856294 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.856299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:52.857806 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:52.857840 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:52.859097 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:52.859107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:52.859401 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:52.860185 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:01:52.860188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:52.860230 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:01:52.860323 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:52.860742 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:52.860756 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:52.860767 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:01:52.860976 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 01:01:52.861491 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":338,"threads_started":1}
I20260929 01:01:52.865017 12836 heartbeater.cc:347] Connected to a master server at 127.3.21.254:46267
I20260929 01:01:52.865445 12836 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:52.866331 12836 heartbeater.cc:511] Master 127.3.21.254:46267 requested a full tablet report, sending...
I20260929 01:01:52.868239 12519 ts_manager.cc:194] Registered new tserver with Master: f79f0297cb7a4dddbc6ce5c6869bdb65 (127.3.21.194:42195)
I20260929 01:01:52.869046 12519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.194:56481
W20260929 01:01:53.220510 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:53.220834 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:53.220999 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:53.228973 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:53.229112 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:53.229285 12839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:53.239746 12839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:53.241111 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:53.243489 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:53.258600 12847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:53.258641 12844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:53.258682 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:53.259406 12839 server_base.cc:1034] running on GCE node
I20260929 01:01:53.260062 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:53.261221 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:53.262465 12839 hybrid_clock.cc:648] HybridClock initialized: now 1790643713262396 us; error 70 us; skew 500 ppm
I20260929 01:01:53.262889 12839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:53.265566 12839 webserver.cc:460] Webserver started at http://127.3.21.195:35251/ using document root <none> and password file <none>
I20260929 01:01:53.266358 12839 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:53.266490 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:53.266857 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:53.269984 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "15e3d46d04ca4a499375f2d7a47c7169"
format_stamp: "Formatted at 2026-09-29 01:01:53 on dist-test-slave-k0b7"
server_key: "46e584436e211033cff2bdeee16a22e2"
server_key_iv: "2f76a3d948a4f5eb2fd0f745fbc3407c"
server_key_version: "encryptionkey@0"
I20260929 01:01:53.271076 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "15e3d46d04ca4a499375f2d7a47c7169"
format_stamp: "Formatted at 2026-09-29 01:01:53 on dist-test-slave-k0b7"
server_key: "46e584436e211033cff2bdeee16a22e2"
server_key_iv: "2f76a3d948a4f5eb2fd0f745fbc3407c"
server_key_version: "encryptionkey@0"
I20260929 01:01:53.277096 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:01:53.281795 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:53.283403 12839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:01:53.283648 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "15e3d46d04ca4a499375f2d7a47c7169"
format_stamp: "Formatted at 2026-09-29 01:01:53 on dist-test-slave-k0b7"
server_key: "46e584436e211033cff2bdeee16a22e2"
server_key_iv: "2f76a3d948a4f5eb2fd0f745fbc3407c"
server_key_version: "encryptionkey@0"
I20260929 01:01:53.283857 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:53.302747 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:53.303764 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:53.304082 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:53.305102 12839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:53.306740 12839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:53.306833 12839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:53.306952 12839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:53.307037 12839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:53.353332 12839 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:40081
I20260929 01:01:53.353430 12965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:40081 every 8 connection(s)
I20260929 01:01:53.354825 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:53.359702 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12839
I20260929 01:01:53.359859 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:01:53.360292 3159 external_mini_cluster.cc:1428] Setting key 6ccfae69440b3a19e5d897c4cb4008c8
I20260929 01:01:53.363999 12705 heartbeater.cc:503] Master 127.3.21.254:46267 was elected leader, sending a full tablet report...
I20260929 01:01:53.370324 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:53.357886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:45793 (local address 127.3.21.254:46267)
0929 01:01:53.358132 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:53.358137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:53.359667 (+ 1530us) server_negotiation.cc:407] Connection header received
0929 01:01:53.361785 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:53.361791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:53.361883 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:53.362013 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:53.363622 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.364603 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.366903 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.367483 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.367616 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:53.369148 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:53.369153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:53.369156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:53.369175 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:53.369207 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:53.369219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:53.369292 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:53.369766 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 01:01:53.369898 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 01:01:53.371196 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:53.358385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:46267 (local address 127.3.21.195:45793)
0929 01:01:53.359483 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:53.359526 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:53.361446 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:53.362419 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:53.362429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:53.362618 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:53.363342 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:53.363373 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.364780 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.364786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:53.366705 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:53.366749 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.367728 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.367733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:53.367971 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:53.368785 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:01:53.368788 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:53.368831 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:01:53.368922 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:53.369335 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:53.369344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:53.369351 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:01:53.369565 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 01:01:53.369921 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":297,"threads_started":1}
I20260929 01:01:53.373462 12966 heartbeater.cc:347] Connected to a master server at 127.3.21.254:46267
I20260929 01:01:53.373831 12966 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:53.374929 12966 heartbeater.cc:511] Master 127.3.21.254:46267 requested a full tablet report, sending...
I20260929 01:01:53.376626 12519 ts_manager.cc:194] Registered new tserver with Master: 15e3d46d04ca4a499375f2d7a47c7169 (127.3.21.195:40081)
I20260929 01:01:53.377501 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:01:53.377645 12519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.195:45793
W20260929 01:01:53.384513 12707 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:53.382659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.3.21.254:46267)
0929 01:01:53.383021 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:53.383029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:53.383288 (+ 259us) server_negotiation.cc:407] Connection header received
0929 01:01:53.383599 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:53.383604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:53.383680 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:53.383845 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:53.384249 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45312: BlockingRecv error: recv got EOF from 127.0.0.1:45312 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:01:53.386233 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12578
I20260929 01:01:53.395408 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:53.387494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.3.21.193:33635)
0929 01:01:53.388207 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:53.388217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:53.388251 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:01:53.388322 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:53.388329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:53.388671 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:53.388839 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:53.389299 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.390338 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.391849 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.392483 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.392621 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:53.393821 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:53.393832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:53.393847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:53.394455 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:53.394557 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:53.394577 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:53.394698 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:53.394896 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 01:01:53.395042 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":245,"threads_started":1}
I20260929 01:01:53.396790 12700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:53.484556 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12578
I20260929 01:01:53.503971 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12709
I20260929 01:01:53.513279 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:53.505003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.3.21.194:42195)
0929 01:01:53.505605 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:53.505617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:53.505656 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:01:53.505722 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:53.505728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:53.506100 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:53.506249 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:53.506708 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.507901 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.509333 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.509936 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.510089 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:53.511557 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:53.511576 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:53.511593 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:53.512323 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:53.512375 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:53.512397 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:53.512519 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:53.512707 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 01:01:53.512880 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
I20260929 01:01:53.514803 12831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:53.602291 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12709
I20260929 01:01:53.617014 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12839
I20260929 01:01:53.626312 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:53.618154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.3.21.195:40081)
0929 01:01:53.618833 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:53.618840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:53.618872 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:01:53.618934 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:53.618940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:53.619281 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:53.619447 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:53.619875 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.621028 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.622550 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:53.623239 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:53.623355 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:53.624510 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:53.624519 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:53.624535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:53.625320 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:53.625370 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:53.625395 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:53.625513 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:53.625704 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 01:01:53.625902 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":380,"threads_started":1}
I20260929 01:01:53.627810 12961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:53.714542 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12839
I20260929 01:01:53.732738 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12488
I20260929 01:01:53.734172 12549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:53.838106 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12488
2026-09-29T01:01:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2688 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T01:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:53Z Disabled control of system clock
I20260929 01:01:53.916425 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36753
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 01:01:54.275501 12985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:54.275879 12985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:54.276036 12985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:54.284145 12985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:54.284317 12985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:54.284389 12985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:54.284471 12985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:54.284557 12985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:54.294445 12985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:54.295883 12985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:54.297766 12985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:54.310527 12990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:54.311204 12993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:54.311451 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:54.311951 12985 server_base.cc:1034] running on GCE node
I20260929 01:01:54.312899 12985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:54.314879 12985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:54.316298 12985 hybrid_clock.cc:648] HybridClock initialized: now 1790643714316244 us; error 45 us; skew 500 ppm
I20260929 01:01:54.316704 12985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:54.319696 12985 webserver.cc:460] Webserver started at http://127.3.21.254:43565/ using document root <none> and password file <none>
I20260929 01:01:54.320621 12985 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:54.320732 12985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:54.321120 12985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:54.326361 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "53a2f42c26634f8daf0a6e575d0691c0"
format_stamp: "Formatted at 2026-09-29 01:01:54 on dist-test-slave-k0b7"
server_key: "bcc6c6d19522bae63b7151e6b7b89f05"
server_key_iv: "709f69edb2941faf1cbd161120e6d1d2"
server_key_version: "encryptionkey@0"
I20260929 01:01:54.327392 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "53a2f42c26634f8daf0a6e575d0691c0"
format_stamp: "Formatted at 2026-09-29 01:01:54 on dist-test-slave-k0b7"
server_key: "bcc6c6d19522bae63b7151e6b7b89f05"
server_key_iv: "709f69edb2941faf1cbd161120e6d1d2"
server_key_version: "encryptionkey@0"
I20260929 01:01:54.334312 12985 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:01:54.339187 12999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:54.341086 12985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:01:54.341336 12985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "53a2f42c26634f8daf0a6e575d0691c0"
format_stamp: "Formatted at 2026-09-29 01:01:54 on dist-test-slave-k0b7"
server_key: "bcc6c6d19522bae63b7151e6b7b89f05"
server_key_iv: "709f69edb2941faf1cbd161120e6d1d2"
server_key_version: "encryptionkey@0"
I20260929 01:01:54.341559 12985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:54.347939 12985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:54.348883 12985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:54.349155 12985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:54.372094 12985 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36753
I20260929 01:01:54.372159 13050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36753 every 8 connection(s)
I20260929 01:01:54.373601 12985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:54.377883 13051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:54.382580 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 12985
I20260929 01:01:54.382818 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:54.383659 3159 external_mini_cluster.cc:1428] Setting key 96ececfbbf0890cc115b7bcc9d92b52f
I20260929 01:01:54.391700 13051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0: Bootstrap starting.
I20260929 01:01:54.393258 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:54.386042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.3.21.254:36753)
0929 01:01:54.387245 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:54.387258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:54.387311 (+ 53us) server_negotiation.cc:407] Connection header received
0929 01:01:54.389160 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:54.389188 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:54.389703 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:54.390189 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:54.391388 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:54.391396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:54.391414 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:54.391463 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:54.391549 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:54.391566 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:54.391800 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:54.391936 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 01:01:54.392279 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":457,"threads_started":1}
I20260929 01:01:54.395449 13051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:54.396696 13051 log.cc:824] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:54.400213 13051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0: No bootstrap required, opened a new log
I20260929 01:01:54.408335 13051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a2f42c26634f8daf0a6e575d0691c0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36753 } }
I20260929 01:01:54.408825 13051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:54.408949 13051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a2f42c26634f8daf0a6e575d0691c0, State: Initialized, Role: FOLLOWER
I20260929 01:01:54.409686 13051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a2f42c26634f8daf0a6e575d0691c0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36753 } }
I20260929 01:01:54.409935 13051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:54.410127 13051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:54.410308 13051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:54.412992 13051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a2f42c26634f8daf0a6e575d0691c0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36753 } }
I20260929 01:01:54.413555 13051 leader_election.cc:302] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53a2f42c26634f8daf0a6e575d0691c0; no voters:
I20260929 01:01:54.414377 13051 leader_election.cc:288] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:54.414578 13056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:54.415316 13056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [term 1 LEADER]: Becoming Leader. State: Replica: 53a2f42c26634f8daf0a6e575d0691c0, State: Running, Role: LEADER
I20260929 01:01:54.416052 13056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a2f42c26634f8daf0a6e575d0691c0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36753 } }
I20260929 01:01:54.416723 13051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:54.419100 13057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53a2f42c26634f8daf0a6e575d0691c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a2f42c26634f8daf0a6e575d0691c0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36753 } } }
I20260929 01:01:54.419744 13057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:54.419612 13058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a2f42c26634f8daf0a6e575d0691c0. Latest consensus state: current_term: 1 leader_uuid: "53a2f42c26634f8daf0a6e575d0691c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a2f42c26634f8daf0a6e575d0691c0" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36753 } } }
I20260929 01:01:54.419950 13058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0 [sys.catalog]: This master's current role is: LEADER
W20260929 01:01:54.430290 13072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:01:54.430516 13072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:01:54.430701 13065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:54.432502 13065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:54.441355 13065 catalog_manager.cc:1347] Generated new cluster ID: 74ae373ef18e4ffc9861511f8a312b97
I20260929 01:01:54.441522 13065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:54.461970 13065 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:54.462587 13065 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:54.471215 13065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53a2f42c26634f8daf0a6e575d0691c0: Generated new TSK 0
I20260929 01:01:54.472402 13065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:01:54.481580 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36753
--builtin_ntp_servers=127.3.21.212:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 01:01:54.861053 13075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:54.861368 13075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:54.861536 13075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:54.870767 13075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:54.870909 13075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:54.871101 13075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:54.881837 13075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:54.883481 13075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:54.885545 13075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:54.901149 13083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:54.901233 13080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:54.901209 13081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:54.901825 13075 server_base.cc:1034] running on GCE node
I20260929 01:01:54.902607 13075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:54.903731 13075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:54.905198 13075 hybrid_clock.cc:648] HybridClock initialized: now 1790643714905210 us; error 114 us; skew 500 ppm
I20260929 01:01:54.905572 13075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:54.908690 13075 webserver.cc:460] Webserver started at http://127.3.21.193:46207/ using document root <none> and password file <none>
I20260929 01:01:54.909546 13075 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:54.909701 13075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:54.910053 13075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:54.913264 13075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "8163e971a95c48be8bafa93344819e3a"
format_stamp: "Formatted at 2026-09-29 01:01:54 on dist-test-slave-k0b7"
server_key: "586a30e45d9abd41025672e08ed232c8"
server_key_iv: "0300802648c8a0ac1ed03a6a6bff53e9"
server_key_version: "encryptionkey@0"
I20260929 01:01:54.914137 13075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "8163e971a95c48be8bafa93344819e3a"
format_stamp: "Formatted at 2026-09-29 01:01:54 on dist-test-slave-k0b7"
server_key: "586a30e45d9abd41025672e08ed232c8"
server_key_iv: "0300802648c8a0ac1ed03a6a6bff53e9"
server_key_version: "encryptionkey@0"
I20260929 01:01:54.920223 13075 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260929 01:01:54.924553 13089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:54.926330 13075 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:01:54.926608 13075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "8163e971a95c48be8bafa93344819e3a"
format_stamp: "Formatted at 2026-09-29 01:01:54 on dist-test-slave-k0b7"
server_key: "586a30e45d9abd41025672e08ed232c8"
server_key_iv: "0300802648c8a0ac1ed03a6a6bff53e9"
server_key_version: "encryptionkey@0"
I20260929 01:01:54.926870 13075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:54.935330 13075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:54.936301 13075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:54.936604 13075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:54.937763 13075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:54.939400 13075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:54.939533 13075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:54.939662 13075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:54.939746 13075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:54.976732 13075 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:44945
I20260929 01:01:54.976787 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:44945 every 8 connection(s)
I20260929 01:01:54.978169 13075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:54.978832 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13075
I20260929 01:01:54.979067 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:54.979858 3159 external_mini_cluster.cc:1428] Setting key 72401ace77b0976b287c58caa4f818e2
I20260929 01:01:54.983048 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36753
--builtin_ntp_servers=127.3.21.212:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 01:01:54.987648 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:54.980977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:56287 (local address 127.3.21.254:36753)
0929 01:01:54.981515 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:54.981521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:54.982822 (+ 1301us) server_negotiation.cc:407] Connection header received
0929 01:01:54.984928 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:54.984934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:54.985011 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:54.985146 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:54.986805 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:54.986811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:54.986816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:54.986838 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:54.986889 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:54.986903 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:54.987045 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:54.987231 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 01:01:54.987304 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":232,"threads_started":1}
I20260929 01:01:54.988873 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:54.981647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36753 (local address 127.3.21.193:56287)
0929 01:01:54.982676 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:54.982713 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:54.984404 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:54.985422 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:54.985432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:54.985542 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:54.986345 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:01:54.986347 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:54.986389 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:01:54.986553 (+ 164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:54.987000 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:54.987011 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:54.987019 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:01:54.987324 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 01:01:54.987636 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":322,"threads_started":1}
I20260929 01:01:54.990675 13202 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36753
I20260929 01:01:54.990993 13202 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:54.991698 13202 heartbeater.cc:511] Master 127.3.21.254:36753 requested a full tablet report, sending...
I20260929 01:01:54.993947 13016 ts_manager.cc:194] Registered new tserver with Master: 8163e971a95c48be8bafa93344819e3a (127.3.21.193:44945)
W20260929 01:01:55.349923 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:55.350313 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:55.350469 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:55.358680 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:55.358807 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:55.358925 13206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:55.370313 13206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:55.372133 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:55.374873 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:55.392017 13211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:55.393218 13212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:55.394697 13214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:55.395198 13206 server_base.cc:1034] running on GCE node
I20260929 01:01:55.396006 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:55.397358 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:55.398775 13206 hybrid_clock.cc:648] HybridClock initialized: now 1790643715398713 us; error 60 us; skew 500 ppm
I20260929 01:01:55.399272 13206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:55.410331 13206 webserver.cc:460] Webserver started at http://127.3.21.194:42099/ using document root <none> and password file <none>
I20260929 01:01:55.411226 13206 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:55.411337 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:55.411738 13206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:55.415099 13206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "afb42a1945c2402b83af0680aa0489ac"
format_stamp: "Formatted at 2026-09-29 01:01:55 on dist-test-slave-k0b7"
server_key: "d5da07990eca6c15740c6320c1ba434e"
server_key_iv: "c868dbccffd7346b5303b0394340e9e7"
server_key_version: "encryptionkey@0"
I20260929 01:01:55.415967 13206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "afb42a1945c2402b83af0680aa0489ac"
format_stamp: "Formatted at 2026-09-29 01:01:55 on dist-test-slave-k0b7"
server_key: "d5da07990eca6c15740c6320c1ba434e"
server_key_iv: "c868dbccffd7346b5303b0394340e9e7"
server_key_version: "encryptionkey@0"
I20260929 01:01:55.422252 13206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:01:55.426767 13220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:55.428247 13206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:01:55.428445 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "afb42a1945c2402b83af0680aa0489ac"
format_stamp: "Formatted at 2026-09-29 01:01:55 on dist-test-slave-k0b7"
server_key: "d5da07990eca6c15740c6320c1ba434e"
server_key_iv: "c868dbccffd7346b5303b0394340e9e7"
server_key_version: "encryptionkey@0"
I20260929 01:01:55.428699 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:55.435122 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:55.435994 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:55.436254 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:55.437217 13206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:55.438817 13206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:55.438937 13206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:55.439028 13206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:55.439090 13206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:55.478374 13206 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:34939
I20260929 01:01:55.478431 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:34939 every 8 connection(s)
I20260929 01:01:55.479800 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:55.490023 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:55.482937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:56473 (local address 127.3.21.254:36753)
0929 01:01:55.483131 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:55.483136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:55.485244 (+ 2108us) server_negotiation.cc:407] Connection header received
0929 01:01:55.487117 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:55.487123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:55.487207 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:55.487320 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:55.489128 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:55.489141 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:55.489145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:55.489166 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:55.489208 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:55.489221 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:55.489389 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:55.489616 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 01:01:55.489723 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:01:55.490237 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13206
I20260929 01:01:55.490429 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:55.491073 3159 external_mini_cluster.cc:1428] Setting key fff02db324e0463f5e26490aeb906964
I20260929 01:01:55.491343 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:55.483587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36753 (local address 127.3.21.194:56473)
0929 01:01:55.484782 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:55.484823 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:55.486778 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:55.487606 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:55.487617 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:55.487724 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:55.488560 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:01:55.488585 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:55.488632 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:01:55.488758 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:55.489357 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:55.489371 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:55.489379 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:01:55.489707 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 01:01:55.490046 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":345,"threads_started":1}
I20260929 01:01:55.493266 13333 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36753
I20260929 01:01:55.493608 13333 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:55.494385 13333 heartbeater.cc:511] Master 127.3.21.254:36753 requested a full tablet report, sending...
I20260929 01:01:55.494625 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36753
--builtin_ntp_servers=127.3.21.212:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 01:01:55.496106 13015 ts_manager.cc:194] Registered new tserver with Master: afb42a1945c2402b83af0680aa0489ac (127.3.21.194:34939)
W20260929 01:01:55.870747 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:55.871066 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:55.871157 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:55.878837 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:55.878963 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:55.879076 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:01:55.888760 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:55.890360 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:55.892338 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:55.907060 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:55.907152 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:55.907013 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:55.907861 13336 server_base.cc:1034] running on GCE node
I20260929 01:01:55.908605 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:55.909732 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:55.910969 13336 hybrid_clock.cc:648] HybridClock initialized: now 1790643715910865 us; error 154 us; skew 500 ppm
I20260929 01:01:55.911502 13336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:01:55.914234 13336 webserver.cc:460] Webserver started at http://127.3.21.195:44819/ using document root <none> and password file <none>
I20260929 01:01:55.915064 13336 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:55.915160 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:55.915500 13336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:55.918869 13336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "d447e3b4559a4961927f73171f70cffe"
format_stamp: "Formatted at 2026-09-29 01:01:55 on dist-test-slave-k0b7"
server_key: "a18f17694f1ed7bfc8b4baa675975c39"
server_key_iv: "c917f101292efe996c7478324050d5ef"
server_key_version: "encryptionkey@0"
I20260929 01:01:55.919795 13336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "d447e3b4559a4961927f73171f70cffe"
format_stamp: "Formatted at 2026-09-29 01:01:55 on dist-test-slave-k0b7"
server_key: "a18f17694f1ed7bfc8b4baa675975c39"
server_key_iv: "c917f101292efe996c7478324050d5ef"
server_key_version: "encryptionkey@0"
I20260929 01:01:55.925714 13336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 01:01:55.930173 13350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:55.932144 13336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:55.932384 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "d447e3b4559a4961927f73171f70cffe"
format_stamp: "Formatted at 2026-09-29 01:01:55 on dist-test-slave-k0b7"
server_key: "a18f17694f1ed7bfc8b4baa675975c39"
server_key_iv: "c917f101292efe996c7478324050d5ef"
server_key_version: "encryptionkey@0"
I20260929 01:01:55.932724 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:55.939288 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:55.940207 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:55.940501 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:55.941612 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:55.943236 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:55.943325 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:55.943483 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:55.943571 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:55.982785 13336 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:36171
I20260929 01:01:55.982825 13462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:36171 every 8 connection(s)
I20260929 01:01:55.984478 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:01:55.992257 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13336
I20260929 01:01:55.992534 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:01:55.993042 3159 external_mini_cluster.cc:1428] Setting key 8ba53d436534fd95e29e908c5fbd7613
I20260929 01:01:55.994287 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:55.987772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:57135 (local address 127.3.21.254:36753)
0929 01:01:55.987950 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:55.987955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:55.989643 (+ 1688us) server_negotiation.cc:407] Connection header received
0929 01:01:55.991503 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:55.991508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:55.991566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:55.991663 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:55.993433 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:55.993439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:55.993444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:55.993465 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:55.993509 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:55.993521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:55.993722 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:55.993917 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 01:01:55.993994 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:01:55.995551 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:55.988277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36753 (local address 127.3.21.195:57135)
0929 01:01:55.989440 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:55.989478 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:55.991162 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:55.991975 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:55.991992 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:55.992265 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:55.993065 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:01:55.993067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:55.993133 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:01:55.993248 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:55.993685 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:55.993695 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:55.993703 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:01:55.994001 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 01:01:55.994324 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":372,"threads_started":1}
I20260929 01:01:55.997277 13202 heartbeater.cc:503] Master 127.3.21.254:36753 was elected leader, sending a full tablet report...
I20260929 01:01:55.997547 13463 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36753
I20260929 01:01:55.997924 13463 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:55.998718 13463 heartbeater.cc:511] Master 127.3.21.254:36753 requested a full tablet report, sending...
I20260929 01:01:56.000511 13015 ts_manager.cc:194] Registered new tserver with Master: d447e3b4559a4961927f73171f70cffe (127.3.21.195:36171)
I20260929 01:01:56.010296 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:01:56.018626 13204 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:01:56.016736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.3.21.254:36753)
0929 01:01:56.016910 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:56.016915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:56.017449 (+ 534us) server_negotiation.cc:407] Connection header received
0929 01:01:56.017582 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:56.017585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:56.017658 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:56.017858 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:56.018223 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34990: BlockingRecv error: recv got EOF from 127.0.0.1:34990 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:01:56.019738 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13075
I20260929 01:01:56.024299 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:56.021244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.3.21.193:44945)
0929 01:01:56.021828 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:56.021835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:56.021862 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:01:56.021923 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:56.021929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:56.022199 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:56.022400 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:56.023472 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:56.023484 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:56.023504 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:56.023525 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:56.023582 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:56.023599 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:56.023741 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:56.023835 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 01:01:56.023929 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
I20260929 01:01:56.025847 13197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:56.111994 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13075
I20260929 01:01:56.132274 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13206
I20260929 01:01:56.137004 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:56.133713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.3.21.194:34939)
0929 01:01:56.134445 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:56.134461 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:56.134590 (+ 129us) server_negotiation.cc:407] Connection header received
0929 01:01:56.134671 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:56.134678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:56.134934 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:56.135137 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:56.136248 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:56.136256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:56.136268 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:56.136288 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:56.136332 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:56.136347 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:56.136503 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:56.136590 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 01:01:56.136697 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":460,"threads_started":1}
I20260929 01:01:56.138419 13328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:56.224643 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13206
I20260929 01:01:56.241382 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13336
I20260929 01:01:56.245337 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:56.242574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.3.21.195:36171)
0929 01:01:56.243238 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:56.243245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:56.243276 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:01:56.243342 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:56.243348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:56.243571 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:56.243714 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:56.244700 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:56.244707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:56.244719 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:01:56.244732 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:56.244777 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:01:56.244794 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:56.244929 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:56.244975 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 01:01:56.245061 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":325,"threads_started":1}
I20260929 01:01:56.247012 13458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:56.338164 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13336
I20260929 01:01:56.355937 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 12985
I20260929 01:01:56.357519 13046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:01:56.462272 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 12985
2026-09-29T01:01:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2627 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:01:56 dist-test-slave-k0b7 krb5kdc[13481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:01:56 dist-test-slave-k0b7 krb5kdc[13481](info): set up 2 sockets
Sep 29 01:01:56 dist-test-slave-k0b7 krb5kdc[13481](info): commencing operation
krb5kdc: starting...
W20260929 01:01:57.612181 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:01:57 dist-test-slave-k0b7 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:01:57.880524 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:42803
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:42803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:58.238679 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:58.239092 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:58.239249 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:58.247186 13494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:01:58.247363 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:58.247469 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:58.247574 13494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:01:58.247660 13494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:01:58.257949 13494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:42803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:42803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:58.259470 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:58.260885 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:58.275681 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:58.275548 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:58.276348 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:58.276878 13494 server_base.cc:1034] running on GCE node
I20260929 01:01:58.277885 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:58.279742 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:58.280977 13494 hybrid_clock.cc:648] HybridClock initialized: now 1790643718280868 us; error 110 us; skew 500 ppm
I20260929 01:01:58.281390 13494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:58 dist-test-slave-k0b7 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643718, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:58.301146 13494 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:01:58.303094 13494 webserver.cc:460] Webserver started at http://127.3.21.254:33433/ using document root <none> and password file <none>
I20260929 01:01:58.303905 13494 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:58.304031 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:58.304371 13494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:58.307826 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "4147f9efc7114ce38da841024a1085be"
format_stamp: "Formatted at 2026-09-29 01:01:58 on dist-test-slave-k0b7"
server_key: "446d5ae5c33463163207811e515719a7"
server_key_iv: "dcd484d10afe0a2837e9b1814bdf07ff"
server_key_version: "encryptionkey@0"
I20260929 01:01:58.308884 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "4147f9efc7114ce38da841024a1085be"
format_stamp: "Formatted at 2026-09-29 01:01:58 on dist-test-slave-k0b7"
server_key: "446d5ae5c33463163207811e515719a7"
server_key_iv: "dcd484d10afe0a2837e9b1814bdf07ff"
server_key_version: "encryptionkey@0"
I20260929 01:01:58.315608 13494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 01:01:58.319748 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:58.321738 13494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:01:58.321969 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "4147f9efc7114ce38da841024a1085be"
format_stamp: "Formatted at 2026-09-29 01:01:58 on dist-test-slave-k0b7"
server_key: "446d5ae5c33463163207811e515719a7"
server_key_iv: "dcd484d10afe0a2837e9b1814bdf07ff"
server_key_version: "encryptionkey@0"
I20260929 01:01:58.322202 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:58.351843 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:58.357573 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:58.357918 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:58.380293 13494 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:42803
I20260929 01:01:58.380400 13560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:42803 every 8 connection(s)
I20260929 01:01:58.381811 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:01:58.386065 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:01:58.387794 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13494
I20260929 01:01:58.388044 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:01:58.388736 3159 external_mini_cluster.cc:1428] Setting key 6e4770cfe91e493c182dab347b7d338d
I20260929 01:01:58.401882 13561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be: Bootstrap starting.
Sep 29 01:01:58 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:58.406741 13561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be: Neither blocks nor log segments found. Creating new log.
I20260929 01:01:58.408332 13561 log.cc:824] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be: Log is configured to *not* fsync() on all Append() calls
I20260929 01:01:58.412775 13561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be: No bootstrap required, opened a new log
I20260929 01:01:58.421010 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:58.390938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.3.21.254:42803)
0929 01:01:58.391824 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:58.391840 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:58.391890 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:01:58.393496 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:58.393522 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:58.393997 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:58.394379 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:58.396436 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:58.397862 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:58.399865 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:58.400791 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:58.400966 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:58.408450 (+ 7484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:58.408459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:58.408502 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:58.408562 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:58.413752 (+ 5190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:58.415268 (+ 1516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:58.415282 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:58.415293 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:58.415593 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:58.416243 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:58.416249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:58.416253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:58.417211 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:58.417784 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:58.418161 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 01:01:58.418997 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":384,"threads_started":1}
I20260929 01:01:58.425530 13561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147f9efc7114ce38da841024a1085be" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42803 } }
I20260929 01:01:58.426158 13561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:01:58.426283 13561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4147f9efc7114ce38da841024a1085be, State: Initialized, Role: FOLLOWER
I20260929 01:01:58.427013 13561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147f9efc7114ce38da841024a1085be" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42803 } }
I20260929 01:01:58.427284 13561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:01:58.427426 13561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:01:58.427623 13561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:01:58.431809 13561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147f9efc7114ce38da841024a1085be" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42803 } }
I20260929 01:01:58.432528 13561 leader_election.cc:302] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4147f9efc7114ce38da841024a1085be; no voters:
I20260929 01:01:58.433262 13561 leader_election.cc:288] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:01:58.433437 13566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:01:58.434171 13566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [term 1 LEADER]: Becoming Leader. State: Replica: 4147f9efc7114ce38da841024a1085be, State: Running, Role: LEADER
I20260929 01:01:58.434859 13566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147f9efc7114ce38da841024a1085be" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42803 } }
I20260929 01:01:58.435789 13561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:01:58.438771 13568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4147f9efc7114ce38da841024a1085be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147f9efc7114ce38da841024a1085be" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42803 } } }
I20260929 01:01:58.439154 13568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:58.438657 13567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4147f9efc7114ce38da841024a1085be. Latest consensus state: current_term: 1 leader_uuid: "4147f9efc7114ce38da841024a1085be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4147f9efc7114ce38da841024a1085be" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42803 } } }
I20260929 01:01:58.439337 13567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be [sys.catalog]: This master's current role is: LEADER
I20260929 01:01:58.440050 13575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:01:58.446777 13575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:01:58.465441 13575 catalog_manager.cc:1347] Generated new cluster ID: 859420f2eaa949a4a89d231466f247f6
I20260929 01:01:58.465601 13575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:01:58.478911 13575 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:01:58.481354 13575 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:01:58.493953 13575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be: Generated new TSK 0
I20260929 01:01:58.495117 13575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:01:58.575847 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42803
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:58.960922 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:58.961287 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:58.961452 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:58.969378 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:58.969565 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:58.969753 13587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:01:58.980350 13587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:58.981809 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:58.983318 13587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:58.999724 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:58.999799 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:58.999842 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:59.000056 13587 server_base.cc:1034] running on GCE node
I20260929 01:01:59.001448 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:59.002664 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:59.003907 13587 hybrid_clock.cc:648] HybridClock initialized: now 1790643719003855 us; error 43 us; skew 500 ppm
I20260929 01:01:59.004400 13587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:59 dist-test-slave-k0b7 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643719, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:59.008288 13587 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:01:59.009936 13587 webserver.cc:460] Webserver started at http://127.3.21.193:44601/ using document root <none> and password file <none>
I20260929 01:01:59.010764 13587 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:59.010908 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:59.011408 13587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:59.014739 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "a5fd2d7e794f42839cf54c24520ad4e4"
format_stamp: "Formatted at 2026-09-29 01:01:59 on dist-test-slave-k0b7"
server_key: "15a1ced18d556b1ee31765d2c8ddc710"
server_key_iv: "13945d41666d30187e1b5d69df81a2ce"
server_key_version: "encryptionkey@0"
I20260929 01:01:59.015594 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "a5fd2d7e794f42839cf54c24520ad4e4"
format_stamp: "Formatted at 2026-09-29 01:01:59 on dist-test-slave-k0b7"
server_key: "15a1ced18d556b1ee31765d2c8ddc710"
server_key_iv: "13945d41666d30187e1b5d69df81a2ce"
server_key_version: "encryptionkey@0"
I20260929 01:01:59.021519 13587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 01:01:59.026539 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:59.028419 13587 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:01:59.028585 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "a5fd2d7e794f42839cf54c24520ad4e4"
format_stamp: "Formatted at 2026-09-29 01:01:59 on dist-test-slave-k0b7"
server_key: "15a1ced18d556b1ee31765d2c8ddc710"
server_key_iv: "13945d41666d30187e1b5d69df81a2ce"
server_key_version: "encryptionkey@0"
I20260929 01:01:59.028846 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:59.054366 13587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:59.059545 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:59.059860 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:59.061007 13587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:59.062918 13587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:59.063001 13587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:59.063094 13587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:59.063138 13587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:59.108368 13587 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:40547
I20260929 01:01:59.108484 13714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:40547 every 8 connection(s)
I20260929 01:01:59.110152 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:01:59.116768 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13587
I20260929 01:01:59.117027 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:01:59.117684 3159 external_mini_cluster.cc:1428] Setting key 3f8be4fba77f4134c93d4ff8e2f7ed3a
Sep 29 01:01:59 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643719, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:59.134470 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:59.113310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:58171 (local address 127.3.21.254:42803)
0929 01:01:59.113792 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:59.113799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:59.115038 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 01:01:59.117065 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:59.117071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:59.117196 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:59.117320 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:59.119115 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.119938 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.121648 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.122073 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.122223 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:59.126047 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:59.126053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:59.126057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:59.126094 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:59.129635 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:59.130923 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:59.130929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:59.130932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:59.131090 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:59.131707 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:59.131712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:59.131714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:59.132116 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:59.132290 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:59.133683 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0929 01:01:59.133952 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":356,"thread_start_us":231,"threads_started":1}
I20260929 01:01:59.135834 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:59.113912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42803 (local address 127.3.21.193:58171)
0929 01:01:59.114855 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:59.114899 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:59.116710 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:59.117590 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:59.117600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:59.118211 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:59.118930 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:59.118950 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.120105 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.120110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:59.121480 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:59.121493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.122338 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.122346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:59.122666 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:59.123383 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:59.123407 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:59.125770 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:59.129841 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:59.129855 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:59.129879 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:59.130687 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:59.131248 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:59.131255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:59.131261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:59.131539 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:59.132312 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:59.132324 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:59.132904 (+ 580us) client_negotiation.cc:769] Sending connection context
0929 01:01:59.133423 (+ 519us) client_negotiation.cc:240] Negotiation successful
0929 01:01:59.134097 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":410,"threads_started":1}
I20260929 01:01:59.137908 13715 heartbeater.cc:347] Connected to a master server at 127.3.21.254:42803
I20260929 01:01:59.138288 13715 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:59.139321 13715 heartbeater.cc:511] Master 127.3.21.254:42803 requested a full tablet report, sending...
I20260929 01:01:59.142227 13526 ts_manager.cc:194] Registered new tserver with Master: a5fd2d7e794f42839cf54c24520ad4e4 (127.3.21.193:40547)
I20260929 01:01:59.143957 13526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:58171
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:01:59.178560 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42803
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:01:59.539023 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:01:59.539340 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:01:59.539529 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:01:59.547947 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:01:59.548122 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:01:59.548301 13721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:01:59.558449 13721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:01:59.559876 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:01:59.561277 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:01:59.575790 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:59.575837 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:01:59.575917 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:01:59.576453 13721 server_base.cc:1034] running on GCE node
I20260929 01:01:59.577262 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:01:59.578292 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:01:59.579519 13721 hybrid_clock.cc:648] HybridClock initialized: now 1790643719579452 us; error 97 us; skew 500 ppm
I20260929 01:01:59.579921 13721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:01:59 dist-test-slave-k0b7 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643719, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:01:59.583756 13721 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:01:59.585626 13721 webserver.cc:460] Webserver started at http://127.3.21.194:39181/ using document root <none> and password file <none>
I20260929 01:01:59.586457 13721 fs_manager.cc:359] Metadata directory not provided
I20260929 01:01:59.586676 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:01:59.587026 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:01:59.590425 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0"
format_stamp: "Formatted at 2026-09-29 01:01:59 on dist-test-slave-k0b7"
server_key: "a2884f0923ba7d6262822d9a61c8f7e3"
server_key_iv: "a9f60bb477863c37535861133a274e06"
server_key_version: "encryptionkey@0"
I20260929 01:01:59.591444 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0"
format_stamp: "Formatted at 2026-09-29 01:01:59 on dist-test-slave-k0b7"
server_key: "a2884f0923ba7d6262822d9a61c8f7e3"
server_key_iv: "a9f60bb477863c37535861133a274e06"
server_key_version: "encryptionkey@0"
I20260929 01:01:59.597869 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 01:01:59.601883 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:59.603605 13721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:01:59.603843 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0"
format_stamp: "Formatted at 2026-09-29 01:01:59 on dist-test-slave-k0b7"
server_key: "a2884f0923ba7d6262822d9a61c8f7e3"
server_key_iv: "a9f60bb477863c37535861133a274e06"
server_key_version: "encryptionkey@0"
I20260929 01:01:59.604059 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:01:59.620990 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:01:59.625761 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:01:59.626091 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:01:59.627311 13721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:01:59.628968 13721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:01:59.629076 13721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:59.629223 13721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:01:59.629303 13721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:01:59.667693 13721 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:38763
I20260929 01:01:59.667760 13848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:38763 every 8 connection(s)
I20260929 01:01:59.669135 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:01:59.674465 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13721
I20260929 01:01:59.674702 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:01:59.675246 3159 external_mini_cluster.cc:1428] Setting key 88a265230990574848a807b04be2ddc9
Sep 29 01:01:59 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643719, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:01:59.693925 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:59.672206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:41925 (local address 127.3.21.254:42803)
0929 01:01:59.672647 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 01:01:59.672654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:01:59.673703 (+ 1049us) server_negotiation.cc:407] Connection header received
0929 01:01:59.676273 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:01:59.676279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:01:59.676375 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:01:59.676497 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:01:59.678373 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.679136 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.681755 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.682260 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.682366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:59.687015 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:01:59.687021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:01:59.687024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:01:59.687101 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:01:59.690138 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:59.691244 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:59.691249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:59.691252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:59.691398 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:01:59.691952 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:01:59.691956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:01:59.691959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:01:59.692364 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:01:59.692484 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:01:59.693472 (+ 988us) server_negotiation.cc:299] Negotiation successful
0929 01:01:59.693669 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":221,"threads_started":1}
I20260929 01:01:59.695545 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:01:59.672781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42803 (local address 127.3.21.194:41925)
0929 01:01:59.673526 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 01:01:59.673575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:01:59.675977 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:01:59.676752 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:01:59.676761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:01:59.677327 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:01:59.678176 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:59.678204 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.679300 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.679309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:59.681527 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:01:59.681546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:01:59.682528 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:01:59.682538 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:01:59.682885 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:01:59.683662 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:01:59.683706 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:01:59.686743 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:01:59.690345 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:59.690359 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:59.690385 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:59.691087 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:59.691528 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:01:59.691535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:01:59.691542 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:01:59.691812 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:01:59.692498 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:01:59.692510 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:01:59.693023 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 01:01:59.693338 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 01:01:59.693939 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":295,"threads_started":1}
I20260929 01:01:59.697312 13849 heartbeater.cc:347] Connected to a master server at 127.3.21.254:42803
I20260929 01:01:59.697661 13849 heartbeater.cc:464] Registering TS with master...
I20260929 01:01:59.698465 13849 heartbeater.cc:511] Master 127.3.21.254:42803 requested a full tablet report, sending...
I20260929 01:01:59.700187 13526 ts_manager.cc:194] Registered new tserver with Master: d2bc39aa407e4a8d8e7e8effb392cdf0 (127.3.21.194:38763)
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:01:59.701517 13526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:41925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:01:59.735755 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42803
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:00.108851 13855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:00.109704 13855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:00.110188 13855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:00.120762 13855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:00.120916 13855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:00.121044 13855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:02:00.133148 13855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:02:00.134898 13855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:00.136813 13855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:00.148501 13715 heartbeater.cc:503] Master 127.3.21.254:42803 was elected leader, sending a full tablet report...
W20260929 01:02:00.154000 13861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:00.154147 13863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:00.154254 13860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:00.155335 13855 server_base.cc:1034] running on GCE node
I20260929 01:02:00.156352 13855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:00.157461 13855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:00.158721 13855 hybrid_clock.cc:648] HybridClock initialized: now 1790643720158650 us; error 65 us; skew 500 ppm
I20260929 01:02:00.159129 13855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:02:00 dist-test-slave-k0b7 krb5kdc[13481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643720, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:00.163228 13855 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:02:00.165347 13855 webserver.cc:460] Webserver started at http://127.3.21.195:45333/ using document root <none> and password file <none>
I20260929 01:02:00.166144 13855 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:00.166307 13855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:00.166725 13855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:00.169966 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "94a51f2468d8470fad6c762baa7d29ab"
format_stamp: "Formatted at 2026-09-29 01:02:00 on dist-test-slave-k0b7"
server_key: "af70932c63fae4e4d3b07e5556247971"
server_key_iv: "7d657b09794f4f7c91c7eea2ce9b7830"
server_key_version: "encryptionkey@0"
I20260929 01:02:00.170924 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "94a51f2468d8470fad6c762baa7d29ab"
format_stamp: "Formatted at 2026-09-29 01:02:00 on dist-test-slave-k0b7"
server_key: "af70932c63fae4e4d3b07e5556247971"
server_key_iv: "7d657b09794f4f7c91c7eea2ce9b7830"
server_key_version: "encryptionkey@0"
I20260929 01:02:00.177275 13855 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 01:02:00.181766 13870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:00.184369 13855 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:02:00.184607 13855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "94a51f2468d8470fad6c762baa7d29ab"
format_stamp: "Formatted at 2026-09-29 01:02:00 on dist-test-slave-k0b7"
server_key: "af70932c63fae4e4d3b07e5556247971"
server_key_iv: "7d657b09794f4f7c91c7eea2ce9b7830"
server_key_version: "encryptionkey@0"
I20260929 01:02:00.184882 13855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:00.213505 13855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:00.218592 13855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:00.218927 13855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:00.220019 13855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:00.221530 13855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:00.221657 13855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:00.221805 13855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:00.221875 13855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:00.260356 13855 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:41767
I20260929 01:02:00.260447 13982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:41767 every 8 connection(s)
I20260929 01:02:00.261722 13855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:00.263414 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 13855
I20260929 01:02:00.263677 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:00.264328 3159 external_mini_cluster.cc:1428] Setting key 855ab90649d0cecef99a547f7c0e535b
Sep 29 01:02:00 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643720, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:02:00.284541 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.265135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:34453 (local address 127.3.21.254:42803)
0929 01:02:00.265718 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.265725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.267076 (+ 1351us) server_negotiation.cc:407] Connection header received
0929 01:02:00.269080 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.269085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.269185 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.269297 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:00.270895 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.271671 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.273400 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.273800 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.273932 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.277585 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:00.277590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:00.277593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:00.277629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:00.280922 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.281869 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.281873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.281875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.282005 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.282661 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.282665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.282668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.282973 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:00.283121 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.284086 (+ 965us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.284260 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":285,"threads_started":1}
I20260929 01:02:00.285627 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.265926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42803 (local address 127.3.21.195:34453)
0929 01:02:00.266937 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:00.266975 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:00.268812 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:00.269542 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:00.269551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:00.270075 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:00.270731 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.270750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.271827 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.271832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.273253 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.273265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.274029 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.274036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.274283 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.275011 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:00.275035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:00.277342 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:00.281098 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:00.281110 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:00.281133 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:00.281672 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:00.282242 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:00.282247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:00.282250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:00.282450 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:00.283104 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:00.283111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:00.283619 (+ 508us) client_negotiation.cc:769] Sending connection context
0929 01:02:00.283962 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 01:02:00.284414 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":394,"threads_started":1}
I20260929 01:02:00.287492 13983 heartbeater.cc:347] Connected to a master server at 127.3.21.254:42803
I20260929 01:02:00.287818 13983 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:00.288659 13983 heartbeater.cc:511] Master 127.3.21.254:42803 requested a full tablet report, sending...
I20260929 01:02:00.290406 13526 ts_manager.cc:194] Registered new tserver with Master: 94a51f2468d8470fad6c762baa7d29ab (127.3.21.195:41767)
I20260929 01:02:00.291337 13526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:34453
I20260929 01:02:00.293793 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:00.313805 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.299070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.3.21.254:42803)
0929 01:02:00.299252 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.299257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.299867 (+ 610us) server_negotiation.cc:407] Connection header received
0929 01:02:00.300092 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.300096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.300174 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.300365 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:00.301817 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.302643 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.303924 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.304349 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.304467 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.307293 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:00.307299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:00.307303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:00.307342 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:00.310726 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.311608 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.311614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.311618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.311764 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.312251 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.312255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.312258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.312690 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:00.312886 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.313294 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.313506 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:02:00.320411 13526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:00.327581 13526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:00.368126 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.342368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.3.21.193:40547)
0929 01:02:00.355296 (+ 12928us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.355312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.355362 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:02:00.355521 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.355535 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.355949 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.356279 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:00.359888 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.361583 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.365045 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.366409 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.366639 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.366940 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.367556 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.367747 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12635,"thread_start_us":12396,"threads_started":1}
I20260929 01:02:00.368454 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.342062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:40547 (local address 127.0.0.1:43692)
0929 01:02:00.353017 (+ 10955us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:00.353093 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:00.353407 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:00.357505 (+ 4098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:00.357519 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:00.357581 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:00.358239 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.358256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.361771 (+ 3515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.361786 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.364568 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.364582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.366925 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.366932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.367263 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.367333 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 01:02:00.367495 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 01:02:00.367832 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10529,"mutex_wait_us":10,"thread_start_us":16542,"threads_started":1}
I20260929 01:02:00.370635 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.343267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:41767 (local address 127.0.0.1:45326)
0929 01:02:00.359895 (+ 16628us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:00.359928 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:00.360254 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:00.361032 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:00.361038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:00.361069 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:00.361523 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.361536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.363687 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.363692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.367248 (+ 3556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.367263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.369697 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.369702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.369917 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.369959 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 01:02:00.370067 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:02:00.370208 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16480,"mutex_wait_us":689,"thread_start_us":9103,"threads_started":1}
I20260929 01:02:00.371002 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.344004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.3.21.195:41767)
0929 01:02:00.359066 (+ 15062us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.359082 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.360226 (+ 1144us) server_negotiation.cc:407] Connection header received
0929 01:02:00.360466 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.360473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.360741 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.361008 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:00.361756 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.363495 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.367422 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.369179 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.369339 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.369570 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.370149 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.370359 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14789,"thread_start_us":15109,"threads_started":1}
I20260929 01:02:00.371206 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.342062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:38763 (local address 127.0.0.1:50166)
0929 01:02:00.358302 (+ 16240us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:00.358339 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:00.358472 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:00.361317 (+ 2845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:00.361322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:00.361359 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:00.361862 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.361881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.364004 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.364009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.366873 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.366895 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.369411 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.369418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.369616 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.369671 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 01:02:00.369784 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 01:02:00.369914 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16063,"mutex_wait_us":514,"thread_start_us":17025,"threads_started":1}
I20260929 01:02:00.371433 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.342323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.3.21.194:38763)
0929 01:02:00.360254 (+ 17931us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.360265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.360297 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:02:00.360413 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.360423 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.360741 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.361008 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:00.362085 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.363844 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.367625 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.369183 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.369333 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.369540 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.370560 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.370817 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17745,"thread_start_us":17939,"threads_started":1}
I20260929 01:02:00.373299 13650 tablet_service.cc:1467] Processing CreateTablet for tablet cf2b95f8a7b7403eb37e62337b923750 (DEFAULT_TABLE table=test-table [id=01095f297d45445eb9feec6d1a64f9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:00.374969 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2b95f8a7b7403eb37e62337b923750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:00.374958 13918 tablet_service.cc:1467] Processing CreateTablet for tablet cf2b95f8a7b7403eb37e62337b923750 (DEFAULT_TABLE table=test-table [id=01095f297d45445eb9feec6d1a64f9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:00.375188 13784 tablet_service.cc:1467] Processing CreateTablet for tablet cf2b95f8a7b7403eb37e62337b923750 (DEFAULT_TABLE table=test-table [id=01095f297d45445eb9feec6d1a64f9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:00.376382 13918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2b95f8a7b7403eb37e62337b923750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:00.376891 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2b95f8a7b7403eb37e62337b923750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:00.398588 14002 tablet_bootstrap.cc:492] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Bootstrap starting.
I20260929 01:02:00.399537 14003 tablet_bootstrap.cc:492] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Bootstrap starting.
I20260929 01:02:00.402616 14002 tablet_bootstrap.cc:654] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:00.403991 14002 log.cc:824] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:00.408294 14003 tablet_bootstrap.cc:654] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:00.408566 14002 tablet_bootstrap.cc:492] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: No bootstrap required, opened a new log
I20260929 01:02:00.408874 14002 ts_tablet_manager.cc:1397] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 01:02:00.410332 14003 log.cc:824] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:00.411791 14004 tablet_bootstrap.cc:492] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Bootstrap starting.
I20260929 01:02:00.415186 14004 tablet_bootstrap.cc:654] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:00.415334 14003 tablet_bootstrap.cc:492] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: No bootstrap required, opened a new log
I20260929 01:02:00.415644 14003 ts_tablet_manager.cc:1397] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260929 01:02:00.416613 14004 log.cc:824] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:00.418853 14002 raft_consensus.cc:348] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.419327 14002 raft_consensus.cc:374] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:00.419422 14002 raft_consensus.cc:729] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a51f2468d8470fad6c762baa7d29ab, State: Initialized, Role: FOLLOWER
I20260929 01:02:00.419617 14004 tablet_bootstrap.cc:492] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: No bootstrap required, opened a new log
I20260929 01:02:00.420001 14004 ts_tablet_manager.cc:1397] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 01:02:00.420111 14002 consensus_queue.cc:260] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.421324 13983 heartbeater.cc:503] Master 127.3.21.254:42803 was elected leader, sending a full tablet report...
I20260929 01:02:00.422132 14002 ts_tablet_manager.cc:1428] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 01:02:00.425153 14003 raft_consensus.cc:348] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.425616 14003 raft_consensus.cc:374] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:00.425717 14003 raft_consensus.cc:729] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fd2d7e794f42839cf54c24520ad4e4, State: Initialized, Role: FOLLOWER
I20260929 01:02:00.426342 14003 consensus_queue.cc:260] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.431679 14004 raft_consensus.cc:348] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.432068 14004 raft_consensus.cc:374] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:00.432286 14004 raft_consensus.cc:729] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bc39aa407e4a8d8e7e8effb392cdf0, State: Initialized, Role: FOLLOWER
I20260929 01:02:00.432992 14004 consensus_queue.cc:260] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.433858 14003 ts_tablet_manager.cc:1428] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260929 01:02:00.434381 13849 heartbeater.cc:503] Master 127.3.21.254:42803 was elected leader, sending a full tablet report...
I20260929 01:02:00.434957 14004 ts_tablet_manager.cc:1428] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260929 01:02:00.513537 13984 tablet.cc:2369] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:00.613384 13716 tablet.cc:2369] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:00.667959 14008 raft_consensus.cc:482] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:00.668275 14008 raft_consensus.cc:504] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.670084 14008 leader_election.cc:288] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5fd2d7e794f42839cf54c24520ad4e4 (127.3.21.193:40547), d2bc39aa407e4a8d8e7e8effb392cdf0 (127.3.21.194:38763)
W20260929 01:02:00.671358 13850 tablet.cc:2369] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:00.678121 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.670840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:44317 (local address 127.3.21.193:40547)
0929 01:02:00.671041 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.671047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.671072 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:02:00.671189 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.671194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.671289 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.671401 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:00.672022 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.673108 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.675855 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.677085 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.677261 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.677342 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.677747 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.677881 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:02:00.678128 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.670519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:40547 (local address 127.3.21.195:44317)
0929 01:02:00.670897 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:00.670924 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:00.671063 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:00.671591 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:00.671596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:00.671623 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:00.671835 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.671848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.673309 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.673314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.675592 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.675620 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.677339 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.677346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.677529 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.677566 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:02:00.677665 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:02:00.677803 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":6}
I20260929 01:02:00.679512 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.670734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:53913 (local address 127.3.21.194:38763)
0929 01:02:00.671060 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.671066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.672847 (+ 1781us) server_negotiation.cc:407] Connection header received
0929 01:02:00.673011 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.673017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.673088 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.673171 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:00.674067 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.675316 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.677028 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.678370 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.678553 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.678626 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.679108 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.679237 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260929 01:02:00.679594 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.671344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:38763 (local address 127.3.21.195:53913)
0929 01:02:00.672720 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:00.672740 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:00.672877 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:00.673208 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:00.673212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:00.673240 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:00.673856 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.673870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.675519 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.675525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.676866 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:00.676877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.678748 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.678753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:00.678928 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.678969 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:02:00.679073 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:02:00.679202 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":23,"thread_start_us":1033,"threads_started":1}
I20260929 01:02:00.679396 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2b95f8a7b7403eb37e62337b923750" candidate_uuid: "94a51f2468d8470fad6c762baa7d29ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" is_pre_election: true
I20260929 01:02:00.679957 13670 raft_consensus.cc:2454] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94a51f2468d8470fad6c762baa7d29ab in term 0.
I20260929 01:02:00.680773 13872 leader_election.cc:302] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94a51f2468d8470fad6c762baa7d29ab, a5fd2d7e794f42839cf54c24520ad4e4; no voters:
I20260929 01:02:00.681245 14008 raft_consensus.cc:2781] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:00.681387 14008 raft_consensus.cc:482] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:00.681491 14008 raft_consensus.cc:3037] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:00.681571 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2b95f8a7b7403eb37e62337b923750" candidate_uuid: "94a51f2468d8470fad6c762baa7d29ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" is_pre_election: true
I20260929 01:02:00.682186 13804 raft_consensus.cc:2454] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94a51f2468d8470fad6c762baa7d29ab in term 0.
I20260929 01:02:00.683930 14008 raft_consensus.cc:504] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.684798 14008 leader_election.cc:288] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [CANDIDATE]: Term 1 election: Requested vote from peers a5fd2d7e794f42839cf54c24520ad4e4 (127.3.21.193:40547), d2bc39aa407e4a8d8e7e8effb392cdf0 (127.3.21.194:38763)
I20260929 01:02:00.685266 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2b95f8a7b7403eb37e62337b923750" candidate_uuid: "94a51f2468d8470fad6c762baa7d29ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fd2d7e794f42839cf54c24520ad4e4"
I20260929 01:02:00.685324 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2b95f8a7b7403eb37e62337b923750" candidate_uuid: "94a51f2468d8470fad6c762baa7d29ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0"
I20260929 01:02:00.685465 13670 raft_consensus.cc:3037] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:00.685557 13804 raft_consensus.cc:3037] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:00.687844 13670 raft_consensus.cc:2454] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94a51f2468d8470fad6c762baa7d29ab in term 1.
I20260929 01:02:00.687851 13804 raft_consensus.cc:2454] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94a51f2468d8470fad6c762baa7d29ab in term 1.
I20260929 01:02:00.688472 13873 leader_election.cc:302] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 94a51f2468d8470fad6c762baa7d29ab, a5fd2d7e794f42839cf54c24520ad4e4, d2bc39aa407e4a8d8e7e8effb392cdf0; no voters:
I20260929 01:02:00.688810 14008 raft_consensus.cc:2781] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:00.689293 14008 raft_consensus.cc:686] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 1 LEADER]: Becoming Leader. State: Replica: 94a51f2468d8470fad6c762baa7d29ab, State: Running, Role: LEADER
I20260929 01:02:00.689836 14008 consensus_queue.cc:237] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } }
I20260929 01:02:00.695278 13526 catalog_manager.cc:5495] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 94a51f2468d8470fad6c762baa7d29ab (127.3.21.195). New cstate: current_term: 1 leader_uuid: "94a51f2468d8470fad6c762baa7d29ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94a51f2468d8470fad6c762baa7d29ab" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41767 } health_report { overall_health: HEALTHY } } }
I20260929 01:02:00.749130 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.742854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.3.21.195:41767)
0929 01:02:00.743067 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.743074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.743242 (+ 168us) server_negotiation.cc:407] Connection header received
0929 01:02:00.743448 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.743453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.743535 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.743726 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:00.744155 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.744952 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.746806 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.747286 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.747403 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.747964 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:00.748593 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:00.748722 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.748852 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.748926 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:02:00.757961 13804 raft_consensus.cc:1261] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 1 FOLLOWER]: Refusing update from remote peer 94a51f2468d8470fad6c762baa7d29ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:00.757961 13670 raft_consensus.cc:1261] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 1 FOLLOWER]: Refusing update from remote peer 94a51f2468d8470fad6c762baa7d29ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:00.759382 14008 consensus_queue.cc:1035] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fd2d7e794f42839cf54c24520ad4e4" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:00.759716 14012 consensus_queue.cc:1035] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bc39aa407e4a8d8e7e8effb392cdf0" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:00.781885 14015 mvcc.cc:205] Tried to move back new op lower bound from 7334476680213757952 to 7334476679949852672. Current Snapshot: MvccSnapshot[applied={T|T < 7334476680213757952}]
I20260929 01:02:00.787052 14017 mvcc.cc:205] Tried to move back new op lower bound from 7334476680213757952 to 7334476679949852672. Current Snapshot: MvccSnapshot[applied={T|T < 7334476680213757952}]
I20260929 01:02:00.796747 13525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:00.797073 13525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:00.800979 13525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4147f9efc7114ce38da841024a1085be: Sending DeleteTablet for 3 replicas of tablet cf2b95f8a7b7403eb37e62337b923750
I20260929 01:02:00.802430 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2b95f8a7b7403eb37e62337b923750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:00 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:43692
I20260929 01:02:00.802547 13784 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2b95f8a7b7403eb37e62337b923750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:00 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:50166
I20260929 01:02:00.803805 13918 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2b95f8a7b7403eb37e62337b923750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:00 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:45326
I20260929 01:02:00.803934 14027 tablet_replica.cc:330] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: stopping tablet replica
I20260929 01:02:00.804382 14027 raft_consensus.cc:2229] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:00.804617 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13587
I20260929 01:02:00.804919 14027 raft_consensus.cc:2258] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:00.807317 14027 ts_tablet_manager.cc:1905] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:00.803493 14026 tablet_replica.cc:330] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: stopping tablet replica
I20260929 01:02:00.810894 14026 raft_consensus.cc:2229] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:00.812969 14026 raft_consensus.cc:2258] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:00.815510 14027 ts_tablet_manager.cc:1918] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:00.815711 14027 log.cc:1196] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/cf2b95f8a7b7403eb37e62337b923750
I20260929 01:02:00.816345 14027 ts_tablet_manager.cc:1939] T cf2b95f8a7b7403eb37e62337b923750 P d2bc39aa407e4a8d8e7e8effb392cdf0: Deleting consensus metadata
I20260929 01:02:00.818019 14026 ts_tablet_manager.cc:1905] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:00.818725 13512 catalog_manager.cc:4841] TS d2bc39aa407e4a8d8e7e8effb392cdf0 (127.3.21.194:38763): tablet cf2b95f8a7b7403eb37e62337b923750 (table test-table [id=01095f297d45445eb9feec6d1a64f9db]) successfully deleted
I20260929 01:02:00.821148 14028 tablet_replica.cc:330] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: stopping tablet replica
I20260929 01:02:00.821606 14028 raft_consensus.cc:2229] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:00.822263 14028 raft_consensus.cc:2258] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:00.827258 14028 ts_tablet_manager.cc:1905] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:00.828169 14026 ts_tablet_manager.cc:1918] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:00.828380 14026 log.cc:1196] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/cf2b95f8a7b7403eb37e62337b923750
I20260929 01:02:00.829010 14026 ts_tablet_manager.cc:1939] T cf2b95f8a7b7403eb37e62337b923750 P a5fd2d7e794f42839cf54c24520ad4e4: Deleting consensus metadata
I20260929 01:02:00.831367 13511 catalog_manager.cc:4841] TS a5fd2d7e794f42839cf54c24520ad4e4 (127.3.21.193:40547): tablet cf2b95f8a7b7403eb37e62337b923750 (table test-table [id=01095f297d45445eb9feec6d1a64f9db]) successfully deleted
I20260929 01:02:00.834057 14028 ts_tablet_manager.cc:1918] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:00.834280 14028 log.cc:1196] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/cf2b95f8a7b7403eb37e62337b923750
I20260929 01:02:00.835012 14028 ts_tablet_manager.cc:1939] T cf2b95f8a7b7403eb37e62337b923750 P 94a51f2468d8470fad6c762baa7d29ab: Deleting consensus metadata
Sep 29 01:02:00 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:02:00.837144 13513 catalog_manager.cc:4841] TS 94a51f2468d8470fad6c762baa7d29ab (127.3.21.195:41767): tablet cf2b95f8a7b7403eb37e62337b923750 (table test-table [id=01095f297d45445eb9feec6d1a64f9db]) successfully deleted
I20260929 01:02:00.843849 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.807351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.3.21.193:40547)
0929 01:02:00.807538 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.807544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.817474 (+ 9930us) server_negotiation.cc:407] Connection header received
0929 01:02:00.818019 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.818024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.818115 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.818210 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:00.822752 (+ 4542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.823583 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.824905 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.825580 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.825717 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.836956 (+ 11239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:00.836978 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:00.836999 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:00.837050 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:00.840564 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.841401 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.841410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.841412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.841559 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.842000 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.842009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.842015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.842429 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:00.842865 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.843190 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.843434 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:02:00.844677 13710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:00.937971 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13587
I20260929 01:02:00.956638 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13721
Sep 29 01:02:00 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:02:00.975279 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:00.957705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.3.21.194:38763)
0929 01:02:00.957915 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:00.957920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:00.957966 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:02:00.958115 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:00.958122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:00.958216 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:00.958350 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:00.959742 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.960857 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.962271 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:00.962968 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:00.963092 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:00.967787 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:00.967810 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:00.967834 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:00.967884 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:00.971767 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.972699 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.972710 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.972714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.972885 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:00.973357 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:00.973362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:00.973366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:00.973870 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:00.974167 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:00.974634 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 01:02:00.974882 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:02:00.976025 13844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:01.071167 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13721
I20260929 01:02:01.095707 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13855
Sep 29 01:02:01 dist-test-slave-k0b7 krb5kdc[13481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:02:01.116218 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:01.096554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.3.21.195:41767)
0929 01:02:01.096753 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:01.096758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:01.096782 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:02:01.096977 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:01.096983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:01.097082 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:01.097347 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:01.098798 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:01.099995 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:01.102050 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:01.103064 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:01.103280 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:01.108511 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:01.108532 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:01.108549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:01.108605 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:01.112898 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:01.113761 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:01.113769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:01.113774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:01.113957 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:01.114380 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:01.114384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:01.114386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:01.114828 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:01.115195 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:01.115498 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 01:02:01.115771 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:02:01.117223 13978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:01.254779 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13855
I20260929 01:02:01.272454 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 13494
I20260929 01:02:01.273712 13556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:01.384104 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 13494
2026-09-29T01:02:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4918 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:01 dist-test-slave-k0b7 krb5kdc[14036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:01 dist-test-slave-k0b7 krb5kdc[14036](info): set up 2 sockets
Sep 29 01:02:01 dist-test-slave-k0b7 krb5kdc[14036](info): commencing operation
krb5kdc: starting...
W20260929 01:02:02.539595 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:02:02 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/rangeradmin_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/rangeradmin_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/rangerlookup_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/rangerlookup_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
I20260929 01:02:02.920104 3159 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmEC8aK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/postgres -l logfile start
2026-09-29 01:02:06.226 UTC [14066] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 01:02:06.226 UTC [14066] LOG: listening on IPv4 address "127.3.21.212", port 40845
2026-09-29 01:02:06.231 UTC [14066] LOG: listening on Unix socket "/tmp/.s.PGSQL.40845"
2026-09-29 01:02:06.261 UTC [14069] LOG: database system was shut down at 2026-09-29 01:02:04 UTC
2026-09-29 01:02:06.266 UTC [14066] LOG: database system is ready to accept connections
I20260929 01:02:07.280037 3159 mini_postgres.cc:96] Postgres bound to 40845
2026-09-29 01:02:07.299 UTC [14077] FATAL: database "slave" does not exist
127.3.21.212:40845 - accepting connections
I20260929 01:02:07.300292 3159 mini_ranger.cc:161] Starting Ranger...
I20260929 01:02:07.331508 3159 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 01:02:08.069447 3159 mini_ranger.cc:87] Created ranger Postgres database
I20260929 01:02:08.069608 3159 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-admin
2026-09-29 01:02:08,403 [I] DB FLAVOR :POSTGRES
2026-09-29 01:02:08,404 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:02:08,404 [I] Checking connection..
2026-09-29 01:02:08,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 01:02:08,700 [I] Checking connection passed.
2026-09-29 01:02:08,700 [I] --------- Verifying version history table ---------
2026-09-29 01:02:08,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:02:08,966 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:02:08,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:02:09,238 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:02:09,238 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 01:02:09,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 01:02:09.513 UTC [14141] WARNING: there is no transaction in progress
2026-09-29 01:02:09,520 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 01:02:09,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:02:09,815 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 01:02:09,815 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:02:09,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 01:02:10,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 01:02:10,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k0b7.c.gcp-upstream.internal','N') ;"
2026-09-29 01:02:10,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:02:10,960 [I] Table x_portal_user does not exist in database ranger
2026-09-29 01:02:10,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:02:11,248 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 01:02:11,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:02:11,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:02:11,797 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 01:02:11,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 01:02:13.081 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.103 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.125 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.146 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.173 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.196 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.267 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.278 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.300 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.314 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.328 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.346 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.360 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.369 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.378 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:13.389 UTC [14249] WARNING: there is no transaction in progress
2026-09-29 01:02:14,354 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 01:02:14,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 01:02:14,639 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 01:02:14,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:02:14,911 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 01:02:14,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:02:15,185 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 01:02:15,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:02:15,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:02:15,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k0b7.c.gcp-upstream.internal';"
2026-09-29 01:02:15,982 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 01:02:15,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 01:02:16,238 [I] DB_PATCHES have already been applied
I20260929 01:02:16.245193 3159 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmEC8aK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 01:02:16.245304 3159 mini_ranger.cc:193] Using host: 127.3.21.212
I20260929 01:02:16.265146 3159 mini_ranger.cc:227] Ranger admin URL: http://127.3.21.212:39949
Sep 29, 2026 1:02:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:02:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:02:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 1:02:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 1:02:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 1:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 1:02:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 01:02:17.474170 3159 mini_ranger.cc:160] Time spent starting Ranger: real 10.174s user 0.000s sys 0.047s
Sep 29, 2026 1:02:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 1:02:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 1:02:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 1:02:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 1:02:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 01:02:42.739460 3159 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/rangerkms_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/rangerkms_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.21.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.21.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 01:02:42 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643762, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 01:02:42.938305 3159 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 01:02:42.962682 3159 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 01:02:44.226461 3159 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 01:02:44.226666 3159 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms
I20260929 01:02:44.226815 3159 mini_ranger_kms.cc:227] Using postgres at 127.3.21.212:40845
2026-09-29 01:02:44,659 [I] DB FLAVOR :POSTGRES
2026-09-29 01:02:44,659 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:02:44,659 [I] Checking connection
2026-09-29 01:02:44,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 01:02:44,930 [I] connection success
2026-09-29 01:02:44,930 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 01:02:44,930 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 01:02:44,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 01:02:45,221 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 01:02:45,221 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:02:45,221 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 01:02:45,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:40845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 01:02:45,527 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 01:02:46.070914 3159 mini_ranger_kms.cc:310] Created kms service
I20260929 01:02:46.258471 3159 mini_ranger_kms.cc:326] Created kudu user
I20260929 01:02:46.313663 3159 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 01:02:46 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643762, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:02:46.774699 3159 mini_ranger_kms.cc:384] Added ranger policy
I20260929 01:02:46.775019 3159 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 01:02:46.775074 3159 mini_ranger_kms.cc:240] Using host: 127.3.21.212
I20260929 01:02:46.795531 3159 mini_ranger_kms.cc:276] Ranger KMS PID: 14430
I20260929 01:02:46.795821 3159 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.21.212:37713
Sep 29, 2026 1:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 1:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 1:02:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 1:02:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 01:02:49 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.21.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:49.283756 3159 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.345s user 0.117s sys 0.339s
I20260929 01:02:49.284005 3159 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 01:02:49.284219 3159 mini_ranger_kms.cc:401] 127.3.21.212:37713/kms/v1/keys
Sep 29 01:02:49 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.21.212@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
Sep 29 01:02:49 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.21.212@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
Sep 29 01:02:52 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:02:52.479691 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37081
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37081
--ranger_config_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:37713/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:52.865272 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:52.865609 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:52.865756 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:52.873651 14480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:52.873787 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:52.873870 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:52.873957 14480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:52.874053 14480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:52.885331 14480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.21.212:37713/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37081
--ranger_config_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:02:52.886912 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:52.888381 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:52.903280 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:52.903294 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:52.904964 14480 server_base.cc:1034] running on GCE node
W20260929 01:02:52.907382 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:52.908664 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:52.910730 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:52.912010 14480 hybrid_clock.cc:648] HybridClock initialized: now 1790643772911943 us; error 65 us; skew 500 ppm
I20260929 01:02:52.912567 14480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:02:52 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:52.916882 14480 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:02:52.918918 14480 webserver.cc:460] Webserver started at http://127.3.21.254:32939/ using document root <none> and password file <none>
I20260929 01:02:52.919683 14480 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:52.919831 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:52.920163 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:02:52 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:02:52.986634 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "44c786921d1f418aa527322761cebcac"
format_stamp: "Formatted at 2026-09-29 01:02:52 on dist-test-slave-k0b7"
server_key: "b486c27f9e408b96f1a0d036bfc11a50"
server_key_iv: "461300aa458a999eb32b8a913cdca02f"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:52.987735 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "44c786921d1f418aa527322761cebcac"
format_stamp: "Formatted at 2026-09-29 01:02:52 on dist-test-slave-k0b7"
server_key: "b486c27f9e408b96f1a0d036bfc11a50"
server_key_iv: "461300aa458a999eb32b8a913cdca02f"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:52.994879 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260929 01:02:53.016599 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:53.018625 14480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:02:53.018857 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "44c786921d1f418aa527322761cebcac"
format_stamp: "Formatted at 2026-09-29 01:02:52 on dist-test-slave-k0b7"
server_key: "b486c27f9e408b96f1a0d036bfc11a50"
server_key_iv: "461300aa458a999eb32b8a913cdca02f"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:53.019129 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:53.037523 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:53.043221 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:53.043582 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:53.070993 14480 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:37081
I20260929 01:02:53.071014 14546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:37081 every 8 connection(s)
I20260929 01:02:53.072678 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:53.078264 14547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:53.081315 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 14480
I20260929 01:02:53.081604 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:02:53.082347 3159 external_mini_cluster.cc:1428] Setting key 9eace855b46aa1bcdb8afa1c95eb307a
I20260929 01:02:53.094172 14547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac: Bootstrap starting.
I20260929 01:02:53.099222 14547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:53.100735 14547 log.cc:824] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:53.104142 14547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac: No bootstrap required, opened a new log
Sep 29 01:02:53 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:02:53.115285 14547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c786921d1f418aa527322761cebcac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37081 } }
I20260929 01:02:53.115916 14547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:53.116082 14547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c786921d1f418aa527322761cebcac, State: Initialized, Role: FOLLOWER
I20260929 01:02:53.116905 14547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c786921d1f418aa527322761cebcac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37081 } }
I20260929 01:02:53.117229 14547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:53.117434 14547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:53.117635 14547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:53.120731 14547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c786921d1f418aa527322761cebcac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37081 } }
I20260929 01:02:53.121408 14547 leader_election.cc:302] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c786921d1f418aa527322761cebcac; no voters:
I20260929 01:02:53.122184 14547 leader_election.cc:288] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:53.122395 14552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:53.124854 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:53.085051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.3.21.254:37081)
0929 01:02:53.096543 (+ 11492us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:53.096567 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:53.097264 (+ 697us) server_negotiation.cc:407] Connection header received
0929 01:02:53.099414 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:53.099463 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:53.099959 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:53.100553 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:53.102240 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.104495 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.106487 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.107721 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.107938 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.113227 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:53.113242 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:53.113271 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:53.113334 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:53.117568 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:53.118709 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:53.118721 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:53.118732 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:53.118942 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:53.119496 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:53.119503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:53.119506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:53.120467 (+ 961us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:53.120821 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:53.121996 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0929 01:02:53.122563 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11119,"thread_start_us":10532,"threads_started":1}
I20260929 01:02:53.131168 14552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [term 1 LEADER]: Becoming Leader. State: Replica: 44c786921d1f418aa527322761cebcac, State: Running, Role: LEADER
I20260929 01:02:53.132807 14547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:53.132726 14552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c786921d1f418aa527322761cebcac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37081 } }
I20260929 01:02:53.137921 14554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c786921d1f418aa527322761cebcac. Latest consensus state: current_term: 1 leader_uuid: "44c786921d1f418aa527322761cebcac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c786921d1f418aa527322761cebcac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37081 } } }
I20260929 01:02:53.138360 14554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:53.138963 14553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c786921d1f418aa527322761cebcac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c786921d1f418aa527322761cebcac" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37081 } } }
I20260929 01:02:53.139241 14553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:53.146536 14558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:53.154701 14558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:53.172413 14558 catalog_manager.cc:1347] Generated new cluster ID: bd287156f1dc4fa89dd62d4a06020794
I20260929 01:02:53.172577 14558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:53.202768 14547 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:02:53.234422 14558 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:53.236910 14558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:53.255836 14558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac: Generated new TSK 0
I20260929 01:02:53.257336 14558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:02:54.172283 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:37713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37081
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:02:54 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 01:02:54.697914 14592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:54.698356 14592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:54.698592 14592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:54.708246 14592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:54.708398 14592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:54.708586 14592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:02:54.719532 14592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.21.212:37713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:02:54.721016 14592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:54.722388 14592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:54.758855 14603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:54.759797 14602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:54.764165 14605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:54.770059 14592 server_base.cc:1034] running on GCE node
I20260929 01:02:54.771361 14592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:54.773439 14592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:54.774742 14592 hybrid_clock.cc:648] HybridClock initialized: now 1790643774774622 us; error 117 us; skew 500 ppm
I20260929 01:02:54.775238 14592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:02:54 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:54.779855 14592 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:02:54.792352 14592 webserver.cc:460] Webserver started at http://127.3.21.193:35985/ using document root <none> and password file <none>
I20260929 01:02:54.793377 14592 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:54.793560 14592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:54.793943 14592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:02:54 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:02:54.813838 14592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6"
format_stamp: "Formatted at 2026-09-29 01:02:54 on dist-test-slave-k0b7"
server_key: "bed214be382853a49ac33dcfcb357dbf"
server_key_iv: "a7cc80927ff447fd4a53d7995bafb32a"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:54.815092 14592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6"
format_stamp: "Formatted at 2026-09-29 01:02:54 on dist-test-slave-k0b7"
server_key: "bed214be382853a49ac33dcfcb357dbf"
server_key_iv: "a7cc80927ff447fd4a53d7995bafb32a"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:54.836511 14592 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.003s sys 0.010s
I20260929 01:02:54.857939 14612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:54.860214 14592 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:02:54.860569 14592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6"
format_stamp: "Formatted at 2026-09-29 01:02:54 on dist-test-slave-k0b7"
server_key: "bed214be382853a49ac33dcfcb357dbf"
server_key_iv: "a7cc80927ff447fd4a53d7995bafb32a"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:54.860968 14592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:54.904811 14592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:54.911279 14592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:54.911764 14592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:54.922981 14592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:54.943570 14592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:54.943801 14592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:02:54.944051 14592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:54.944202 14592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:54 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:02:55.111483 14592 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:36219
I20260929 01:02:55.111500 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:36219 every 8 connection(s)
I20260929 01:02:55.113595 14592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:55.120800 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 14592
I20260929 01:02:55.121022 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:55.121862 3159 external_mini_cluster.cc:1428] Setting key 94f83e941202798eb0e917e5e11f5795
Sep 29 01:02:55 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:02:55.158633 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.117146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:54567 (local address 127.3.21.254:37081)
0929 01:02:55.135231 (+ 18085us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:55.135240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:55.135275 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:02:55.135875 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:55.135883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:55.136002 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:55.136115 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:55.138894 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.140291 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.143275 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.144042 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.144178 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.149719 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:55.149727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:55.149732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:55.149781 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:55.154170 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:55.155533 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:55.155540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:55.155544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:55.155726 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:55.156352 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:55.156358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:55.156362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:55.156728 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:55.156875 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:55.157953 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0929 01:02:55.158206 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17876,"thread_start_us":11262,"threads_started":1}
I20260929 01:02:55.160943 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.117489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37081 (local address 127.3.21.193:54567)
0929 01:02:55.129470 (+ 11981us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:55.129556 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:55.135404 (+ 5848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:55.136816 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:55.136831 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:55.137595 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:55.138614 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.138649 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.140559 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.140565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.143026 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.143048 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.144610 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.144618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.144963 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.145999 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:55.146037 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:55.149350 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:55.154448 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:55.154459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:55.154522 (+ 63us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:55.155329 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:55.155902 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:55.155907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:55.155911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:55.156204 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:55.156981 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:55.156990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:55.157502 (+ 512us) client_negotiation.cc:769] Sending connection context
0929 01:02:55.158869 (+ 1367us) client_negotiation.cc:240] Negotiation successful
0929 01:02:55.159475 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11509,"thread_start_us":15530,"threads_started":1}
I20260929 01:02:55.168843 14725 heartbeater.cc:347] Connected to a master server at 127.3.21.254:37081
I20260929 01:02:55.169263 14725 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:55.170308 14725 heartbeater.cc:511] Master 127.3.21.254:37081 requested a full tablet report, sending...
I20260929 01:02:55.173163 14512 ts_manager.cc:194] Registered new tserver with Master: 1f5cc2bcfb9d49f9a3475f9ab21c8af6 (127.3.21.193:36219)
I20260929 01:02:55.175705 14512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:54567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:02:55.199661 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:37713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37081
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:02:55 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
W20260929 01:02:55.663717 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:55.664077 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:55.664230 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:55.672508 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:55.672645 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:55.672847 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:02:55.683753 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.21.212:37713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:02:55.685159 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:55.686566 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:55.701489 14745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:55.701550 14746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:55.701581 14748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:55.702337 14731 server_base.cc:1034] running on GCE node
I20260929 01:02:55.703022 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:55.704085 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:55.705387 14731 hybrid_clock.cc:648] HybridClock initialized: now 1790643775705336 us; error 43 us; skew 500 ppm
I20260929 01:02:55.705824 14731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:02:55 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:55.710140 14731 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:02:55.712049 14731 webserver.cc:460] Webserver started at http://127.3.21.194:39981/ using document root <none> and password file <none>
I20260929 01:02:55.712833 14731 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:55.712998 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:55.713361 14731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:02:55 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:02:55.732142 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "4fdf03b2ee0d43e38255d0aaa731bf24"
format_stamp: "Formatted at 2026-09-29 01:02:55 on dist-test-slave-k0b7"
server_key: "2eb7d852389e044ac170f15919b0954f"
server_key_iv: "e94ee66b384249a27482933dea5d135f"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:55.733587 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "4fdf03b2ee0d43e38255d0aaa731bf24"
format_stamp: "Formatted at 2026-09-29 01:02:55 on dist-test-slave-k0b7"
server_key: "2eb7d852389e044ac170f15919b0954f"
server_key_iv: "e94ee66b384249a27482933dea5d135f"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:55.740885 14731 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260929 01:02:55.757681 14755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:55.759400 14731 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:02:55.759621 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "4fdf03b2ee0d43e38255d0aaa731bf24"
format_stamp: "Formatted at 2026-09-29 01:02:55 on dist-test-slave-k0b7"
server_key: "2eb7d852389e044ac170f15919b0954f"
server_key_iv: "e94ee66b384249a27482933dea5d135f"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:55.759929 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:55.786556 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:55.791436 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:55.791786 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:55.792857 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:55.795076 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:55.795197 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:55.795377 14731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:55.795454 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:55.837124 14731 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:41123
I20260929 01:02:55.837227 14867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:41123 every 8 connection(s)
I20260929 01:02:55.838622 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:55.845024 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 14731
I20260929 01:02:55.845357 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:55.845857 3159 external_mini_cluster.cc:1428] Setting key 049df27812b42e60eb5adb73339abf65
Sep 29 01:02:55 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:02:55.863997 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.841838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:33619 (local address 127.3.21.254:37081)
0929 01:02:55.842417 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:55.842422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:55.843525 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 01:02:55.845792 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:55.845799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:55.845915 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:55.846068 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:55.848069 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.849207 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.851401 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.852152 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.852302 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.856791 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:55.856798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:55.856801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:55.856846 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:55.859969 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:55.861095 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:55.861101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:55.861103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:55.861243 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:55.861845 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:55.861848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:55.861850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:55.862132 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:55.862325 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:55.863349 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0929 01:02:55.863543 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":361,"threads_started":1}
I20260929 01:02:55.865535 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.842369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37081 (local address 127.3.21.194:33619)
0929 01:02:55.843345 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:55.843387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:55.845407 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:55.846414 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:55.846428 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:55.846970 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:55.847877 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.847901 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.849411 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.849415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.851221 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.851244 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.852426 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.852432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.852679 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.853766 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:55.853800 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:55.856447 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:55.860165 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:55.860179 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:55.860208 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:55.860920 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:55.861385 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:55.861393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:55.861399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:55.861721 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:55.862275 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:55.862287 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:55.862834 (+ 547us) client_negotiation.cc:769] Sending connection context
0929 01:02:55.863176 (+ 342us) client_negotiation.cc:240] Negotiation successful
0929 01:02:55.863883 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":319,"threads_started":1}
I20260929 01:02:55.867451 14868 heartbeater.cc:347] Connected to a master server at 127.3.21.254:37081
I20260929 01:02:55.867841 14868 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:55.868649 14868 heartbeater.cc:511] Master 127.3.21.254:37081 requested a full tablet report, sending...
I20260929 01:02:55.870431 14512 ts_manager.cc:194] Registered new tserver with Master: 4fdf03b2ee0d43e38255d0aaa731bf24 (127.3.21.194:41123)
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:02:55.871426 14512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:33619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:02:55.907441 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:37713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37081
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:02:56.180624 14725 heartbeater.cc:503] Master 127.3.21.254:37081 was elected leader, sending a full tablet report...
W20260929 01:02:56.284564 14874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:56.284902 14874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:56.285053 14874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:56.293150 14874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:56.293284 14874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:56.293460 14874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:02:56.304641 14874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.21.212:37713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:02:56.306134 14874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:56.307581 14874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:56.323963 14879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:56.324023 14880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:56.324414 14882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:56.325736 14874 server_base.cc:1034] running on GCE node
I20260929 01:02:56.326773 14874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:56.328270 14874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:56.329566 14874 hybrid_clock.cc:648] HybridClock initialized: now 1790643776329457 us; error 75 us; skew 500 ppm
I20260929 01:02:56.329986 14874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:02:56 dist-test-slave-k0b7 krb5kdc[14036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:56.335683 14874 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:02:56.338090 14874 webserver.cc:460] Webserver started at http://127.3.21.195:39461/ using document root <none> and password file <none>
I20260929 01:02:56.338991 14874 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:56.339185 14874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:56.339568 14874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:02:56 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:02:56.358956 14874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "4941c7aeb9004bf18de31c0b37d6e627"
format_stamp: "Formatted at 2026-09-29 01:02:56 on dist-test-slave-k0b7"
server_key: "d18207ed807ccf635c043f950d699ed6"
server_key_iv: "e5853bd96de5c2f06689dc39a4fb3964"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:56.359917 14874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "4941c7aeb9004bf18de31c0b37d6e627"
format_stamp: "Formatted at 2026-09-29 01:02:56 on dist-test-slave-k0b7"
server_key: "d18207ed807ccf635c043f950d699ed6"
server_key_iv: "e5853bd96de5c2f06689dc39a4fb3964"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:56.367326 14874 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 01:02:56.388010 14889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:56.390069 14874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:02:56.390391 14874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "4941c7aeb9004bf18de31c0b37d6e627"
format_stamp: "Formatted at 2026-09-29 01:02:56 on dist-test-slave-k0b7"
server_key: "d18207ed807ccf635c043f950d699ed6"
server_key_iv: "e5853bd96de5c2f06689dc39a4fb3964"
server_key_version: "kuduclusterkey@0"
I20260929 01:02:56.390708 14874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:56.405886 14874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:56.411705 14874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:56.412074 14874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:56.413131 14874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:56.414826 14874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:56.414902 14874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:56.415035 14874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:56.415095 14874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:56.458608 14874 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37619
I20260929 01:02:56.458673 15001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37619 every 8 connection(s)
I20260929 01:02:56.459956 14874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:56.465193 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 14874
I20260929 01:02:56.465485 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:56.465916 3159 external_mini_cluster.cc:1428] Setting key fba82dc7aa56e549762e15bf2743b4fc
Sep 29 01:02:56 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:02:56.484084 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.462857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:36105 (local address 127.3.21.254:37081)
0929 01:02:56.463475 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.463480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.464659 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 01:02:56.467294 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.467302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.467388 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.467611 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:56.469418 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.470244 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.472108 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.472631 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.472796 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.476582 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:56.476588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:56.476591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:56.476626 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:56.480454 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:56.481407 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:56.481416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:56.481419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:56.481579 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:56.482158 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:56.482165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:56.482169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:56.482573 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:56.482697 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.483641 (+ 944us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.483826 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":281,"threads_started":1}
I20260929 01:02:56.485075 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.463421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37081 (local address 127.3.21.195:36105)
0929 01:02:56.464482 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.464539 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.466782 (+ 2243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.467892 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.467902 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.468475 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:56.469245 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.469267 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.470408 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.470414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.471933 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.471947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.472892 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.472902 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.473179 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.473989 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:56.474016 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:56.476310 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:56.480616 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:56.480629 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:56.480651 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:56.481262 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:56.481691 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:56.481695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:56.481698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:56.482010 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:56.482688 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:56.482695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:56.483051 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 01:02:56.483319 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 01:02:56.483734 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":437,"threads_started":1}
I20260929 01:02:56.486833 15002 heartbeater.cc:347] Connected to a master server at 127.3.21.254:37081
I20260929 01:02:56.487121 15002 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:56.487815 15002 heartbeater.cc:511] Master 127.3.21.254:37081 requested a full tablet report, sending...
I20260929 01:02:56.489472 14512 ts_manager.cc:194] Registered new tserver with Master: 4941c7aeb9004bf18de31c0b37d6e627 (127.3.21.195:37619)
I20260929 01:02:56.490363 14512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:36105
I20260929 01:02:56.494973 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:56.514608 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.499878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.3.21.254:37081)
0929 01:02:56.500089 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.500094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.500417 (+ 323us) server_negotiation.cc:407] Connection header received
0929 01:02:56.500554 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.500558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.500639 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.500738 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:56.502430 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.503296 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.504892 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.505391 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.505498 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.508518 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:56.508523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:56.508527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:56.508562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:56.511343 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:56.512442 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:56.512447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:56.512449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:56.512591 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:56.513034 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:56.513039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:56.513042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:56.513463 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:56.513680 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.514089 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.514282 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 01:02:56.520202 14512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:56.528645 14512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:56.571887 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.545106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.3.21.193:36219)
0929 01:02:56.559635 (+ 14529us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.559646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.559683 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:02:56.559795 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.559804 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.560109 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.560371 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:56.561391 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.564227 (+ 2836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.568666 (+ 4439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.570076 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.570370 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.570647 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.571231 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.571455 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14327,"thread_start_us":14160,"threads_started":1}
I20260929 01:02:56.572857 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.544945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36219 (local address 127.0.0.1:47608)
0929 01:02:56.558395 (+ 13450us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.558467 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.558775 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.560545 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.560554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.560588 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:56.561149 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.561168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.564438 (+ 3270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.564444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.568464 (+ 4020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.568486 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.570529 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.570537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.570844 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.570917 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 01:02:56.572256 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0929 01:02:56.572507 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13015,"mutex_wait_us":152,"spinlock_wait_cycles":32000,"thread_start_us":14585,"threads_started":1}
I20260929 01:02:56.573314 15016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.545030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:41123 (local address 127.0.0.1:56892)
0929 01:02:56.558473 (+ 13443us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.558515 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.558776 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.561818 (+ 3042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.561824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.561860 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:56.562329 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.562342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.564677 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.564688 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.568484 (+ 3796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.568495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.572445 (+ 3950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.572451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.572644 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.572676 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 01:02:56.572749 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:02:56.572903 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13276,"mutex_wait_us":201,"spinlock_wait_cycles":21504,"thread_start_us":14039,"threads_started":1}
I20260929 01:02:56.574671 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.555109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.3.21.194:41123)
0929 01:02:56.561092 (+ 5983us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.561104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.561138 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:02:56.561256 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.561268 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.561569 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.562591 (+ 1022us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:56.562854 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.564473 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.570744 (+ 6271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.572152 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.573677 (+ 1525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.573932 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.574022 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.574226 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5748,"thread_start_us":5621,"threads_started":1}
I20260929 01:02:56.575507 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.560654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.3.21.195:37619)
0929 01:02:56.565600 (+ 4946us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.565612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.566081 (+ 469us) server_negotiation.cc:407] Connection header received
0929 01:02:56.566317 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.566325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.566735 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.566963 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:56.568092 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.569517 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.572403 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.573859 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.574025 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.574995 (+ 970us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.575064 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.575208 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4753,"thread_start_us":4575,"threads_started":1}
I20260929 01:02:56.576256 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.561193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37619 (local address 127.0.0.1:33106)
0929 01:02:56.565947 (+ 4754us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.565963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.566144 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.567271 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.567276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.567299 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:56.567868 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.567885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.569704 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.569708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.571911 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.571931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.574079 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.574085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.574274 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.574319 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:02:56.574405 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:02:56.574575 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4623,"thread_start_us":4450,"threads_started":1}
I20260929 01:02:56.577957 14660 tablet_service.cc:1467] Processing CreateTablet for tablet 6ddb8b029ee741c28fa330b3e7bb88d3 (DEFAULT_TABLE table=test-table [id=52abcb4e17cd4f67873e3d6ea7833f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:56.578269 14803 tablet_service.cc:1467] Processing CreateTablet for tablet 6ddb8b029ee741c28fa330b3e7bb88d3 (DEFAULT_TABLE table=test-table [id=52abcb4e17cd4f67873e3d6ea7833f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:56.579313 14660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ddb8b029ee741c28fa330b3e7bb88d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:56.579900 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ddb8b029ee741c28fa330b3e7bb88d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:56.582247 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 6ddb8b029ee741c28fa330b3e7bb88d3 (DEFAULT_TABLE table=test-table [id=52abcb4e17cd4f67873e3d6ea7833f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:56.583797 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ddb8b029ee741c28fa330b3e7bb88d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:56.592876 15021 tablet_bootstrap.cc:492] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Bootstrap starting.
I20260929 01:02:56.595268 15022 tablet_bootstrap.cc:492] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Bootstrap starting.
I20260929 01:02:56.598286 15021 tablet_bootstrap.cc:654] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:56.599536 15022 tablet_bootstrap.cc:654] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:56.600163 15023 tablet_bootstrap.cc:492] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Bootstrap starting.
I20260929 01:02:56.601048 15022 log.cc:824] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:56.602185 15021 log.cc:824] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:56.604003 15023 tablet_bootstrap.cc:654] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:56.605484 15022 tablet_bootstrap.cc:492] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: No bootstrap required, opened a new log
I20260929 01:02:56.605705 15023 log.cc:824] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:56.605857 15022 ts_tablet_manager.cc:1397] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260929 01:02:56.608680 15023 tablet_bootstrap.cc:492] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: No bootstrap required, opened a new log
I20260929 01:02:56.608990 15023 ts_tablet_manager.cc:1397] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 01:02:56.613656 15021 tablet_bootstrap.cc:492] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: No bootstrap required, opened a new log
I20260929 01:02:56.614019 15021 ts_tablet_manager.cc:1397] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.003s
I20260929 01:02:56.616756 15022 raft_consensus.cc:348] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.617185 15022 raft_consensus.cc:374] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:56.617317 15022 raft_consensus.cc:729] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5cc2bcfb9d49f9a3475f9ab21c8af6, State: Initialized, Role: FOLLOWER
I20260929 01:02:56.617992 15022 consensus_queue.cc:260] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.619913 15022 ts_tablet_manager.cc:1428] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260929 01:02:56.621181 15023 raft_consensus.cc:348] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.621687 15023 raft_consensus.cc:374] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:56.621798 15023 raft_consensus.cc:729] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4941c7aeb9004bf18de31c0b37d6e627, State: Initialized, Role: FOLLOWER
I20260929 01:02:56.622517 15023 consensus_queue.cc:260] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.624310 15002 heartbeater.cc:503] Master 127.3.21.254:37081 was elected leader, sending a full tablet report...
I20260929 01:02:56.625010 15023 ts_tablet_manager.cc:1428] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260929 01:02:56.627017 15021 raft_consensus.cc:348] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.627609 15021 raft_consensus.cc:374] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:56.627733 15021 raft_consensus.cc:729] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdf03b2ee0d43e38255d0aaa731bf24, State: Initialized, Role: FOLLOWER
I20260929 01:02:56.628448 15021 consensus_queue.cc:260] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.629947 14868 heartbeater.cc:503] Master 127.3.21.254:37081 was elected leader, sending a full tablet report...
I20260929 01:02:56.630659 15021 ts_tablet_manager.cc:1428] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Time spent starting tablet: real 0.017s user 0.015s sys 0.004s
W20260929 01:02:56.711903 15003 tablet.cc:2369] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:56.753049 15029 raft_consensus.cc:482] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:56.753412 15029 raft_consensus.cc:504] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.755432 15029 leader_election.cc:288] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5cc2bcfb9d49f9a3475f9ab21c8af6 (127.3.21.193:36219), 4941c7aeb9004bf18de31c0b37d6e627 (127.3.21.195:37619)
I20260929 01:02:56.763335 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.755767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36219 (local address 127.3.21.194:33321)
0929 01:02:56.756873 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.756890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.757068 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.757420 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.757425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.757448 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:56.758364 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.758375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.759420 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.759429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.761014 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.761029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.762292 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.762298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.762539 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.762582 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:02:56.762694 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 01:02:56.762854 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":17,"thread_start_us":725,"threads_started":1}
I20260929 01:02:56.763278 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.756442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:33321 (local address 127.3.21.193:36219)
0929 01:02:56.756663 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.756669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.756993 (+ 324us) server_negotiation.cc:407] Connection header received
0929 01:02:56.757222 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.757226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.757286 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.757397 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:56.758541 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.759267 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.761203 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.762066 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.762274 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.762364 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.762750 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.762934 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:02:56.764079 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.756113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:39025 (local address 127.3.21.195:37619)
0929 01:02:56.756351 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.756358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.757087 (+ 729us) server_negotiation.cc:407] Connection header received
0929 01:02:56.757277 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.757287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.757351 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.757447 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:56.758613 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.759319 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.761561 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.762634 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.762845 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.762936 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.763590 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.763754 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:02:56.764969 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.755919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37619 (local address 127.3.21.194:39025)
0929 01:02:56.756979 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.756994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.757139 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.757460 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.757464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.757485 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:56.758402 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.758412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.759460 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.759467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.761389 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.761402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.763111 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.763119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.763369 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.763414 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:02:56.764320 (+ 906us) client_negotiation.cc:240] Negotiation successful
0929 01:02:56.764498 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":18,"thread_start_us":765,"threads_started":1}
I20260929 01:02:56.764999 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ddb8b029ee741c28fa330b3e7bb88d3" candidate_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" is_pre_election: true
I20260929 01:02:56.765722 14680 raft_consensus.cc:2454] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdf03b2ee0d43e38255d0aaa731bf24 in term 0.
I20260929 01:02:56.766551 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ddb8b029ee741c28fa330b3e7bb88d3" candidate_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4941c7aeb9004bf18de31c0b37d6e627" is_pre_election: true
I20260929 01:02:56.766692 14759 leader_election.cc:302] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5cc2bcfb9d49f9a3475f9ab21c8af6, 4fdf03b2ee0d43e38255d0aaa731bf24; no voters:
I20260929 01:02:56.767109 14957 raft_consensus.cc:2454] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdf03b2ee0d43e38255d0aaa731bf24 in term 0.
I20260929 01:02:56.767167 15029 raft_consensus.cc:2781] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:56.767416 15029 raft_consensus.cc:482] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:56.767549 15029 raft_consensus.cc:3037] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:56.770605 15029 raft_consensus.cc:504] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.771384 15029 leader_election.cc:288] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5cc2bcfb9d49f9a3475f9ab21c8af6 (127.3.21.193:36219), 4941c7aeb9004bf18de31c0b37d6e627 (127.3.21.195:37619)
I20260929 01:02:56.772053 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ddb8b029ee741c28fa330b3e7bb88d3" candidate_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4941c7aeb9004bf18de31c0b37d6e627"
I20260929 01:02:56.772056 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ddb8b029ee741c28fa330b3e7bb88d3" candidate_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6"
I20260929 01:02:56.772315 14680 raft_consensus.cc:3037] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:56.772430 14957 raft_consensus.cc:3037] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:56.776198 14680 raft_consensus.cc:2454] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fdf03b2ee0d43e38255d0aaa731bf24 in term 1.
I20260929 01:02:56.776198 14957 raft_consensus.cc:2454] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fdf03b2ee0d43e38255d0aaa731bf24 in term 1.
I20260929 01:02:56.776752 14759 leader_election.cc:302] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5cc2bcfb9d49f9a3475f9ab21c8af6, 4fdf03b2ee0d43e38255d0aaa731bf24; no voters:
I20260929 01:02:56.777082 15029 raft_consensus.cc:2781] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:56.777627 15029 raft_consensus.cc:686] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 1 LEADER]: Becoming Leader. State: Replica: 4fdf03b2ee0d43e38255d0aaa731bf24, State: Running, Role: LEADER
I20260929 01:02:56.778154 15029 consensus_queue.cc:237] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } }
I20260929 01:02:56.783713 14512 catalog_manager.cc:5495] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fdf03b2ee0d43e38255d0aaa731bf24 (127.3.21.194). New cstate: current_term: 1 leader_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fdf03b2ee0d43e38255d0aaa731bf24" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:56.817602 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.811205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.3.21.194:41123)
0929 01:02:56.811459 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.811464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.811504 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:02:56.811602 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.811605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.811716 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.811873 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:56.812465 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.813191 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.815028 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.815463 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.815566 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.816115 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:56.816806 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:56.816925 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.817070 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.817143 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 01:02:56.827855 14957 raft_consensus.cc:1261] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 1 FOLLOWER]: Refusing update from remote peer 4fdf03b2ee0d43e38255d0aaa731bf24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:56.827857 14680 raft_consensus.cc:1261] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 1 FOLLOWER]: Refusing update from remote peer 4fdf03b2ee0d43e38255d0aaa731bf24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:56.829092 15029 consensus_queue.cc:1035] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4941c7aeb9004bf18de31c0b37d6e627" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:56.829530 15032 consensus_queue.cc:1035] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5cc2bcfb9d49f9a3475f9ab21c8af6" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:02:56.841163 14869 tablet.cc:2369] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:56.858120 15037 mvcc.cc:205] Tried to move back new op lower bound from 7334476909873639424 to 7334476909688573952. Current Snapshot: MvccSnapshot[applied={T|T < 7334476909873639424}]
I20260929 01:02:56.865693 14511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:56.866086 14511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:56.870432 14511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44c786921d1f418aa527322761cebcac: Sending DeleteTablet for 3 replicas of tablet 6ddb8b029ee741c28fa330b3e7bb88d3
I20260929 01:02:56.871987 14660 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ddb8b029ee741c28fa330b3e7bb88d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:56 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:47608
I20260929 01:02:56.871996 14803 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ddb8b029ee741c28fa330b3e7bb88d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:56 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:56892
I20260929 01:02:56.872805 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ddb8b029ee741c28fa330b3e7bb88d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:56 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:33106
I20260929 01:02:56.875010 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 14592
I20260929 01:02:56.879315 15045 tablet_replica.cc:330] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: stopping tablet replica
I20260929 01:02:56.879956 15045 raft_consensus.cc:2229] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:56.880661 15045 raft_consensus.cc:2258] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:56.883461 15045 ts_tablet_manager.cc:1905] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:56.885965 15044 tablet_replica.cc:330] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: stopping tablet replica
I20260929 01:02:56.886613 15044 raft_consensus.cc:2229] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:56.887557 15044 raft_consensus.cc:2258] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:56.888484 15046 tablet_replica.cc:330] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: stopping tablet replica
I20260929 01:02:56.888944 15046 raft_consensus.cc:2229] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:56.889500 15046 raft_consensus.cc:2258] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:56.891783 15046 ts_tablet_manager.cc:1905] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:56.894779 15044 ts_tablet_manager.cc:1905] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:56.895376 15045 ts_tablet_manager.cc:1918] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:56.895563 15045 log.cc:1196] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/6ddb8b029ee741c28fa330b3e7bb88d3
I20260929 01:02:56.896191 15045 ts_tablet_manager.cc:1939] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 1f5cc2bcfb9d49f9a3475f9ab21c8af6: Deleting consensus metadata
I20260929 01:02:56.898329 14499 catalog_manager.cc:4841] TS 1f5cc2bcfb9d49f9a3475f9ab21c8af6 (127.3.21.193:36219): tablet 6ddb8b029ee741c28fa330b3e7bb88d3 (table test-table [id=52abcb4e17cd4f67873e3d6ea7833f93]) successfully deleted
Sep 29 01:02:56 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:02:56.904083 15046 ts_tablet_manager.cc:1918] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:56.904119 15044 ts_tablet_manager.cc:1918] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:56.904294 15044 log.cc:1196] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/6ddb8b029ee741c28fa330b3e7bb88d3
I20260929 01:02:56.904284 15046 log.cc:1196] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/6ddb8b029ee741c28fa330b3e7bb88d3
I20260929 01:02:56.904839 15046 ts_tablet_manager.cc:1939] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4941c7aeb9004bf18de31c0b37d6e627: Deleting consensus metadata
I20260929 01:02:56.904841 15044 ts_tablet_manager.cc:1939] T 6ddb8b029ee741c28fa330b3e7bb88d3 P 4fdf03b2ee0d43e38255d0aaa731bf24: Deleting consensus metadata
I20260929 01:02:56.906981 14497 catalog_manager.cc:4841] TS 4fdf03b2ee0d43e38255d0aaa731bf24 (127.3.21.194:41123): tablet 6ddb8b029ee741c28fa330b3e7bb88d3 (table test-table [id=52abcb4e17cd4f67873e3d6ea7833f93]) successfully deleted
I20260929 01:02:56.907446 14497 catalog_manager.cc:4841] TS 4941c7aeb9004bf18de31c0b37d6e627 (127.3.21.195:37619): tablet 6ddb8b029ee741c28fa330b3e7bb88d3 (table test-table [id=52abcb4e17cd4f67873e3d6ea7833f93]) successfully deleted
I20260929 01:02:56.912295 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.876350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.3.21.193:36219)
0929 01:02:56.876722 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.876730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.888326 (+ 11596us) server_negotiation.cc:407] Connection header received
0929 01:02:56.888710 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.888717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.888824 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.888904 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:56.891019 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.892207 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.894397 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.895136 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.895253 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.903463 (+ 8210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:56.903491 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:56.903516 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:56.903585 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:56.908585 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:56.909553 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:56.909567 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:56.909571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:56.909767 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:56.910275 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:56.910282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:56.910286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:56.910871 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:56.911219 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:56.911557 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 01:02:56.911849 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260929 01:02:56.913218 14720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:57.012576 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 14592
I20260929 01:02:57.037019 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 14731
Sep 29 01:02:57 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:02:57.057801 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:57.037988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.3.21.194:41123)
0929 01:02:57.038193 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:57.038200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:57.038225 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:02:57.038394 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:57.038401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:57.038612 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:57.038758 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:57.040239 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.041639 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.043496 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.044271 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.044438 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.050244 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:57.050268 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:57.050287 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:57.050336 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:57.054014 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:57.055131 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:57.055138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:57.055141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:57.055340 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:57.055882 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:57.055887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:57.055889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:57.056525 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:57.056803 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:57.057260 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 01:02:57.057459 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:02:57.058681 14863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:57.176826 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 14731
I20260929 01:02:57.195881 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 14874
Sep 29 01:02:57 dist-test-slave-k0b7 krb5kdc[14036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:02:57.212630 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:57.196857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.3.21.195:37619)
0929 01:02:57.197069 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:57.197077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:57.197099 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:57.197393 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:57.197397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:57.197471 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:57.197582 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:57.198975 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.199690 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.200987 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.201390 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.201530 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.206299 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:57.206315 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:57.206331 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:57.206373 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:57.209506 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:57.210346 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:57.210355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:57.210357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:57.210576 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:57.210964 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:57.210971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:57.210974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:57.211369 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:57.211692 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:57.212010 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 01:02:57.212264 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:02:57.213287 14997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:57.308378 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 14874
I20260929 01:02:57.326184 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 14480
I20260929 01:02:57.327565 14542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:02:57.447911 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 14480
I20260929 01:02:57.471163 3159 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 01:02:57.807971 3159 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 01:02:57.808211 3159 mini_ranger.cc:66] Stopping Ranger...
I20260929 01:02:58.155640 3159 mini_ranger.cc:68] Stopped Ranger
2026-09-29 01:02:58.155 UTC [14066] LOG: received smart shutdown request
2026-09-29 01:02:58.164 UTC [14066] LOG: background worker "logical replication launcher" (PID 14075) exited with exit code 1
2026-09-29 01:02:58.164 UTC [14070] LOG: shutting down
2026-09-29 01:02:58.397 UTC [14066] LOG: database system is shut down
2026-09-29T01:02:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57144 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:58 dist-test-slave-k0b7 krb5kdc[15060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:58 dist-test-slave-k0b7 krb5kdc[15060](info): set up 2 sockets
Sep 29 01:02:58 dist-test-slave-k0b7 krb5kdc[15060](info): commencing operation
krb5kdc: starting...
W20260929 01:02:59.719367 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:02:59 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/rangeradmin_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/rangeradmin_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/rangerlookup_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/rangerlookup_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
I20260929 01:03:00.111111 3159 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmEC8aK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/postgres -l logfile start
2026-09-29 01:03:03.609 UTC [15090] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 01:03:03.609 UTC [15090] LOG: listening on IPv4 address "127.3.21.212", port 35925
2026-09-29 01:03:03.615 UTC [15090] LOG: listening on Unix socket "/tmp/.s.PGSQL.35925"
2026-09-29 01:03:03.640 UTC [15093] LOG: database system was shut down at 2026-09-29 01:03:01 UTC
2026-09-29 01:03:03.645 UTC [15090] LOG: database system is ready to accept connections
I20260929 01:03:04.646672 3159 mini_postgres.cc:96] Postgres bound to 35925
2026-09-29 01:03:04.665 UTC [15101] FATAL: database "slave" does not exist
127.3.21.212:35925 - accepting connections
I20260929 01:03:04.666399 3159 mini_ranger.cc:161] Starting Ranger...
I20260929 01:03:04.697057 3159 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 01:03:05.420271 3159 mini_ranger.cc:87] Created ranger Postgres database
I20260929 01:03:05.420390 3159 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-admin
2026-09-29 01:03:05,744 [I] DB FLAVOR :POSTGRES
2026-09-29 01:03:05,744 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:03:05,745 [I] Checking connection..
2026-09-29 01:03:05,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 01:03:06,027 [I] Checking connection passed.
2026-09-29 01:03:06,027 [I] --------- Verifying version history table ---------
2026-09-29 01:03:06,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:06,287 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:03:06,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:06,560 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:03:06,560 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 01:03:06,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 01:03:06.830 UTC [15165] WARNING: there is no transaction in progress
2026-09-29 01:03:06,839 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 01:03:06,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:07,129 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 01:03:07,130 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:03:07,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 01:03:07,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 01:03:07,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k0b7.c.gcp-upstream.internal','N') ;"
2026-09-29 01:03:07,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:03:08,214 [I] Table x_portal_user does not exist in database ranger
2026-09-29 01:03:08,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:03:08,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 01:03:08,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:03:08,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:03:09,085 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 01:03:09,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 01:03:10.457 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.484 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.509 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.532 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.557 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.583 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.666 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.677 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.702 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.716 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.731 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.748 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.762 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.774 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.784 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:10.795 UTC [15273] WARNING: there is no transaction in progress
2026-09-29 01:03:11,815 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 01:03:11,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 01:03:12,081 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 01:03:12,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:03:12,356 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 01:03:12,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:03:12,633 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 01:03:12,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:03:12,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:03:13,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k0b7.c.gcp-upstream.internal';"
2026-09-29 01:03:13,401 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 01:03:13,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 01:03:13,659 [I] DB_PATCHES have already been applied
I20260929 01:03:13.665307 3159 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmEC8aK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 01:03:13.665419 3159 mini_ranger.cc:193] Using host: 127.3.21.212
I20260929 01:03:13.682215 3159 mini_ranger.cc:227] Ranger admin URL: http://127.3.21.212:59253
Sep 29, 2026 1:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 1:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 1:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 1:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 1:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 01:03:14.876528 3159 mini_ranger.cc:160] Time spent starting Ranger: real 10.210s user 0.002s sys 0.044s
Sep 29, 2026 1:03:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 1:03:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 1:03:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 1:03:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 1:03:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 01:03:40.118080 3159 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.21.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.21.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/rangerkms_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.21.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/rangerkms_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.21.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.21.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.21.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/HTTP_127.3.21.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 01:03:40 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 01:03:40.320533 3159 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 01:03:40.346735 3159 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 01:03:41.700699 3159 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 01:03:41.700836 3159 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms
I20260929 01:03:41.700891 3159 mini_ranger_kms.cc:227] Using postgres at 127.3.21.212:35925
2026-09-29 01:03:42,079 [I] DB FLAVOR :POSTGRES
2026-09-29 01:03:42,079 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:03:42,080 [I] Checking connection
2026-09-29 01:03:42,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 01:03:42,346 [I] connection success
2026-09-29 01:03:42,346 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 01:03:42,347 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 01:03:42,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 01:03:42,613 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 01:03:42,613 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:03:42,614 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 01:03:42,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.21.212:35925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 01:03:42,942 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 01:03:43.571674 3159 mini_ranger_kms.cc:310] Created kms service
I20260929 01:03:43.798020 3159 mini_ranger_kms.cc:326] Created kudu user
I20260929 01:03:43.847883 3159 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 01:03:43 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:44.292402 3159 mini_ranger_kms.cc:384] Added ranger policy
I20260929 01:03:44.292757 3159 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmEC8aK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmEC8aK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 01:03:44.292825 3159 mini_ranger_kms.cc:240] Using host: 127.3.21.212
I20260929 01:03:44.312328 3159 mini_ranger_kms.cc:276] Ranger KMS PID: 15454
I20260929 01:03:44.312497 3159 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.21.212:44037
Sep 29, 2026 1:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 1:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 1:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 1:03:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 01:03:46.703369 3159 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.383s user 0.087s sys 0.310s
I20260929 01:03:46.703650 3159 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 01:03:46.703866 3159 mini_ranger_kms.cc:401] 127.3.21.212:44037/kms/v1/keys
Sep 29 01:03:46 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.21.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:03:47 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.21.212@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
Sep 29 01:03:47 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.21.212@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
Sep 29 01:03:50 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:03:50.200096 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:39169
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:39169
--ranger_config_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:50.548238 15504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:50.548640 15504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:50.548816 15504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:50.556491 15504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:03:50.556704 15504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:50.556819 15504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:50.556926 15504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:03:50.557011 15504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:03:50.568722 15504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.21.212:44037/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:39169
--ranger_config_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:50.570297 15504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:50.571769 15504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:50.586865 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:50.586948 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:50.586907 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:50.588442 15504 server_base.cc:1034] running on GCE node
I20260929 01:03:50.589457 15504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:50.591184 15504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:50.592489 15504 hybrid_clock.cc:648] HybridClock initialized: now 1790643830592433 us; error 57 us; skew 500 ppm
I20260929 01:03:50.592965 15504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:50 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:50.596715 15504 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:03:50.598675 15504 webserver.cc:460] Webserver started at http://127.3.21.254:42421/ using document root <none> and password file <none>
I20260929 01:03:50.599416 15504 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:50.599567 15504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:50.599931 15504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:50 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:50.660049 15504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "04add40b01aa4a4da33f527eb2bb2277"
format_stamp: "Formatted at 2026-09-29 01:03:50 on dist-test-slave-k0b7"
server_key: "5db342618cf4bb8e8961cff3325f58b1"
server_key_iv: "2980de9d2c008b52f6aeac255eac917b"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:50.661224 15504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "04add40b01aa4a4da33f527eb2bb2277"
format_stamp: "Formatted at 2026-09-29 01:03:50 on dist-test-slave-k0b7"
server_key: "5db342618cf4bb8e8961cff3325f58b1"
server_key_iv: "2980de9d2c008b52f6aeac255eac917b"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:50.667699 15504 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 01:03:50.688725 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:50.690627 15504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:03:50.690878 15504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "04add40b01aa4a4da33f527eb2bb2277"
format_stamp: "Formatted at 2026-09-29 01:03:50 on dist-test-slave-k0b7"
server_key: "5db342618cf4bb8e8961cff3325f58b1"
server_key_iv: "2980de9d2c008b52f6aeac255eac917b"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:50.691159 15504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:50.712087 15504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:50.717218 15504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:50.717600 15504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:50.740105 15504 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:39169
I20260929 01:03:50.740154 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:39169 every 8 connection(s)
I20260929 01:03:50.741678 15504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:03:50.746027 15571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:50.746994 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 15504
I20260929 01:03:50.747244 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:03:50.747967 3159 external_mini_cluster.cc:1428] Setting key 7799684ba6de91a4a34be5d91875729b
I20260929 01:03:50.761422 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Bootstrap starting.
Sep 29 01:03:50 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:03:50.766152 15571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:50.767696 15571 log.cc:824] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:50.772754 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: No bootstrap required, opened a new log
I20260929 01:03:50.777602 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:50.750372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.3.21.254:39169)
0929 01:03:50.751426 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:50.751443 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:50.751501 (+ 58us) server_negotiation.cc:407] Connection header received
0929 01:03:50.753375 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:50.753410 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:50.754024 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:50.754548 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:50.755978 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:50.758031 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:50.759730 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:50.760788 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:50.760993 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:50.767045 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:50.767065 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:50.767136 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:50.767241 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:50.772144 (+ 4903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:50.773187 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:50.773197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:50.773209 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:50.773474 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:50.773922 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:50.773928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:50.773931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:50.774767 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:50.775122 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:50.775503 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 01:03:50.775983 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":439,"threads_started":1}
I20260929 01:03:50.783805 15571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:50.784245 15571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:50.784332 15571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04add40b01aa4a4da33f527eb2bb2277, State: Initialized, Role: FOLLOWER
I20260929 01:03:50.784962 15571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:50.785198 15571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:03:50.785315 15571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:03:50.785449 15571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:50.788178 15571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:50.788766 15571 leader_election.cc:302] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04add40b01aa4a4da33f527eb2bb2277; no voters:
I20260929 01:03:50.789489 15571 leader_election.cc:288] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:03:50.789647 15576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:50.791095 15576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 LEADER]: Becoming Leader. State: Replica: 04add40b01aa4a4da33f527eb2bb2277, State: Running, Role: LEADER
I20260929 01:03:50.792033 15576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:50.792634 15571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:03:50.795573 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04add40b01aa4a4da33f527eb2bb2277. Latest consensus state: current_term: 1 leader_uuid: "04add40b01aa4a4da33f527eb2bb2277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } } }
I20260929 01:03:50.795531 15577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04add40b01aa4a4da33f527eb2bb2277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } } }
I20260929 01:03:50.796224 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:50.796290 15577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:50.797835 15585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:03:50.806429 15585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:03:50.808826 15571 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:03:50.836573 15585 catalog_manager.cc:1347] Generated new cluster ID: e6ee62ec5f4b4dc084523f78113b0b0a
I20260929 01:03:50.836743 15585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:03:50.848418 15585 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:03:50.849937 15585 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:03:50.881302 15585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Generated new TSK 0
I20260929 01:03:50.883147 15585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:03:51.667851 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39169
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:03:51 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643831, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 01:03:52.114923 15616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:52.115219 15616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:52.115394 15616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:52.125089 15616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:52.125206 15616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:52.125380 15616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:03:52.139055 15616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.21.212:44037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:52.140844 15616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:52.143004 15616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:52.162824 15626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:52.162757 15627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:52.176250 15629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:52.176545 15616 server_base.cc:1034] running on GCE node
I20260929 01:03:52.177435 15616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:52.178754 15616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:52.180040 15616 hybrid_clock.cc:648] HybridClock initialized: now 1790643832179965 us; error 62 us; skew 500 ppm
I20260929 01:03:52.180442 15616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:52 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:52.184386 15616 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:03:52.186461 15616 webserver.cc:460] Webserver started at http://127.3.21.193:40369/ using document root <none> and password file <none>
I20260929 01:03:52.187346 15616 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:52.187518 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:52.187868 15616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:52 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:52.207352 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "e0698cdab2044a2089e58a836f3c10e9"
format_stamp: "Formatted at 2026-09-29 01:03:52 on dist-test-slave-k0b7"
server_key: "06a72e7c098380074475548be0a109a2"
server_key_iv: "9cd565b6f730094243de28c80c0024b7"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:52.208704 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "e0698cdab2044a2089e58a836f3c10e9"
format_stamp: "Formatted at 2026-09-29 01:03:52 on dist-test-slave-k0b7"
server_key: "06a72e7c098380074475548be0a109a2"
server_key_iv: "9cd565b6f730094243de28c80c0024b7"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:52.228111 15616 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.012s
I20260929 01:03:52.267112 15636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:52.270563 15616 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.005s sys 0.012s
I20260929 01:03:52.270807 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "e0698cdab2044a2089e58a836f3c10e9"
format_stamp: "Formatted at 2026-09-29 01:03:52 on dist-test-slave-k0b7"
server_key: "06a72e7c098380074475548be0a109a2"
server_key_iv: "9cd565b6f730094243de28c80c0024b7"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:52.271095 15616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:52.314141 15616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:52.319998 15616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:52.320365 15616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:52.331923 15616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:52.334204 15616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:52.334342 15616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:52.334508 15616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:52.334580 15616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 01:03:52.438935 15616 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:41483
I20260929 01:03:52.438949 15748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:41483 every 8 connection(s)
I20260929 01:03:52.440747 15616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:03:52.451042 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 15616
I20260929 01:03:52.451282 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:03:52.452263 3159 external_mini_cluster.cc:1428] Setting key 2c8d045623a9aa2d6e5f7ea1ca8b2388
Sep 29 01:03:52 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:03:52.486238 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:52.445673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:41967 (local address 127.3.21.254:39169)
0929 01:03:52.447721 (+ 2048us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:52.447728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:52.451718 (+ 3990us) server_negotiation.cc:407] Connection header received
0929 01:03:52.455836 (+ 4118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:52.455842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:52.455950 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:52.456054 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:52.465589 (+ 9535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:52.466804 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:52.469529 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:52.470243 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:52.470356 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:52.478857 (+ 8501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:52.478867 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:52.478872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:52.478941 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:52.482189 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:52.483283 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:52.483289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:52.483292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:52.483457 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:52.484033 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:52.484038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:52.484041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:52.484396 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:52.484544 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:52.485631 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0929 01:03:52.485880 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1897,"thread_start_us":1841,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:03:52.488637 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:52.444984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39169 (local address 127.3.21.193:41967)
0929 01:03:52.451446 (+ 6462us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:52.451540 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:52.455336 (+ 3796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:52.462949 (+ 7613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:52.462967 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:52.463714 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:52.464629 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:52.464662 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:52.467058 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:52.467064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:52.469313 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:52.469333 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:52.470775 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:52.470782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:52.471118 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:52.475409 (+ 4291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:52.475454 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:52.478471 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:52.482398 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:52.482408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:52.482430 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:52.483115 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:52.483615 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:52.483620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:52.483624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:52.483894 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:52.484669 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:52.484681 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:52.485181 (+ 500us) client_negotiation.cc:769] Sending connection context
0929 01:03:52.486613 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0929 01:03:52.487203 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5904,"thread_start_us":16355,"threads_started":1}
I20260929 01:03:52.490680 15749 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39169
I20260929 01:03:52.491019 15749 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:03:52.491870 15749 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:52.494521 15536 ts_manager.cc:194] Registered new tserver with Master: e0698cdab2044a2089e58a836f3c10e9 (127.3.21.193:41483)
I20260929 01:03:52.496546 15536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:41967
Sep 29 01:03:52 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643831, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:03:52.531491 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39169
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:03:52 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643831, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
W20260929 01:03:53.052709 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:53.053256 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:53.053468 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:53.065389 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:53.065631 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:53.065889 15755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:03:53.082135 15755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.21.212:44037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:53.084363 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:53.086205 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:53.121649 15763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:53.128270 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:53.128597 15762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:53.131091 15755 server_base.cc:1034] running on GCE node
I20260929 01:03:53.132789 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:53.140736 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:53.142412 15755 hybrid_clock.cc:648] HybridClock initialized: now 1790643833142287 us; error 132 us; skew 500 ppm
I20260929 01:03:53.143011 15755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:53 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:53.148201 15755 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:03:53.191043 15755 webserver.cc:460] Webserver started at http://127.3.21.194:35483/ using document root <none> and password file <none>
I20260929 01:03:53.192368 15755 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:53.192648 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:53.193210 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:53 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:53.222754 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "5d87bbc936d44907911dbd84a961bb1b"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "996c0cdccc90f4bc6aa887fb5a3379d2"
server_key_iv: "77f44b532c302445f2e047ac2cac5696"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:53.223902 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "5d87bbc936d44907911dbd84a961bb1b"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "996c0cdccc90f4bc6aa887fb5a3379d2"
server_key_iv: "77f44b532c302445f2e047ac2cac5696"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:53.255080 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.031s user 0.008s sys 0.020s
I20260929 01:03:53.287252 15778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:53.289572 15755 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.014s sys 0.000s
I20260929 01:03:53.289825 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "5d87bbc936d44907911dbd84a961bb1b"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "996c0cdccc90f4bc6aa887fb5a3379d2"
server_key_iv: "77f44b532c302445f2e047ac2cac5696"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:53.290113 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:53.380434 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:53.386415 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:53.386821 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:53.388154 15755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:53.390153 15755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:53.390275 15755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:53.390411 15755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:53.390502 15755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:53.455096 15755 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:38277
I20260929 01:03:53.455099 15891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:38277 every 8 connection(s)
I20260929 01:03:53.456758 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:03:53.457250 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 15755
I20260929 01:03:53.457449 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:03:53.457938 3159 external_mini_cluster.cc:1428] Setting key b34626f6e6bade964082add1701953f8
Sep 29 01:03:53 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:03:53.494475 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:53.462797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:55241 (local address 127.3.21.254:39169)
0929 01:03:53.466575 (+ 3778us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:53.466583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:53.472051 (+ 5468us) server_negotiation.cc:407] Connection header received
0929 01:03:53.473551 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:53.473558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:53.473673 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:53.473802 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:53.476031 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:53.477245 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:53.480420 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:53.481050 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:53.481200 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:53.486450 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:53.486459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:53.486464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:53.486533 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:53.490317 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:53.491387 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:53.491395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:53.491398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:53.491557 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:53.492257 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:53.492263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:53.492266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:53.492668 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:53.492833 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:53.493841 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0929 01:03:53.494100 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3612,"thread_start_us":3701,"threads_started":1}
I20260929 01:03:53.495702 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:53.462105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39169 (local address 127.3.21.194:55241)
0929 01:03:53.471148 (+ 9043us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:53.471218 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:53.473180 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:53.474160 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:53.474174 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:53.474875 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:53.475776 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:53.475807 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:53.477485 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:53.477494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:53.480127 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:53.480196 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:53.481335 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:53.481344 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:53.481697 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:53.482674 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:53.482712 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:53.485994 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:53.490605 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:53.490614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:53.490649 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:53.491181 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:53.491687 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:53.491691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:53.491750 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:53.492114 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:53.492838 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:53.492848 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:53.493380 (+ 532us) client_negotiation.cc:769] Sending connection context
0929 01:03:53.493690 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 01:03:53.494241 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8678,"thread_start_us":8435,"threads_started":1}
I20260929 01:03:53.497430 15892 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39169
I20260929 01:03:53.497800 15892 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:53.498646 15892 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:53.500404 15536 ts_manager.cc:194] Registered new tserver with Master: 5d87bbc936d44907911dbd84a961bb1b (127.3.21.194:38277)
I20260929 01:03:53.500859 15749 heartbeater.cc:503] Master 127.3.21.254:39169 was elected leader, sending a full tablet report...
I20260929 01:03:53.501574 15536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:55241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:03:53.530588 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39169
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:53.877367 15898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:53.877782 15898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:53.877959 15898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:53.885564 15898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:53.885746 15898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:53.885980 15898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:03:53.896950 15898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.21.212:44037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:53.898294 15898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:53.900030 15898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:53.914724 15903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:53.914731 15906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:53.915017 15904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:53.915630 15898 server_base.cc:1034] running on GCE node
I20260929 01:03:53.916302 15898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:53.917516 15898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:53.918759 15898 hybrid_clock.cc:648] HybridClock initialized: now 1790643833918706 us; error 48 us; skew 500 ppm
I20260929 01:03:53.919129 15898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:53 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:53.922623 15898 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:03:53.924311 15898 webserver.cc:460] Webserver started at http://127.3.21.195:41767/ using document root <none> and password file <none>
I20260929 01:03:53.925083 15898 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:53.925240 15898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:53.925575 15898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:53 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:53.944043 15898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "4232363cd54b405abc58fb64de664bd2"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "9eb6c2fb478eb76470b2285e46b203df"
server_key_iv: "951cdc7caedc996e3de942421deea840"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:53.945065 15898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "4232363cd54b405abc58fb64de664bd2"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "9eb6c2fb478eb76470b2285e46b203df"
server_key_iv: "951cdc7caedc996e3de942421deea840"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:53.951999 15898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:03:53.970099 15913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:53.971967 15898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:03:53.972186 15898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "4232363cd54b405abc58fb64de664bd2"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "9eb6c2fb478eb76470b2285e46b203df"
server_key_iv: "951cdc7caedc996e3de942421deea840"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:53.972427 15898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:54.001526 15898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:54.006981 15898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:54.007362 15898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:54.008538 15898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:54.010046 15898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:54.010155 15898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:54.010301 15898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:54.010375 15898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:54.045573 15898 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:38031
I20260929 01:03:54.045595 16025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:38031 every 8 connection(s)
I20260929 01:03:54.046929 15898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:03:54.048363 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 15898
I20260929 01:03:54.048539 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:03:54.049053 3159 external_mini_cluster.cc:1428] Setting key b49ce8d16da49d4e5a9802746c9829f5
Sep 29 01:03:54 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:03:54.069089 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:54.049879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:60441 (local address 127.3.21.254:39169)
0929 01:03:54.050421 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:54.050426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:54.051689 (+ 1263us) server_negotiation.cc:407] Connection header received
0929 01:03:54.053609 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:54.053614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:54.053737 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:54.053870 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:54.055423 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.056162 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.057785 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.058212 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.058355 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:54.061734 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:54.061740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:54.061743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:54.061777 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:54.065457 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.066370 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.066383 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.066386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.066598 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.067118 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.067123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.067127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.067472 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:54.067645 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:54.068415 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 01:03:54.068664 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":240,"threads_started":1}
I20260929 01:03:54.070189 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:54.050594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39169 (local address 127.3.21.195:60441)
0929 01:03:54.051524 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:54.051587 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:54.053340 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:54.054103 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:54.054111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:54.054640 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:54.055242 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:54.055260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.056316 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.056333 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:54.057632 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:54.057652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.058453 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.058459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:54.058723 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:54.059435 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:54.059459 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:54.061485 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:54.065631 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:54.065650 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:54.065667 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:54.066217 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:54.066725 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:54.066730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:54.066734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:54.066978 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:54.067617 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:54.067627 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:54.068006 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 01:03:54.068304 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 01:03:54.068890 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":387,"threads_started":1}
I20260929 01:03:54.071979 16026 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39169
I20260929 01:03:54.072338 16026 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:54.073159 16026 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:54.075024 15535 ts_manager.cc:194] Registered new tserver with Master: 4232363cd54b405abc58fb64de664bd2 (127.3.21.195:38031)
I20260929 01:03:54.075984 15535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:60441
I20260929 01:03:54.078322 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:03:54.080283 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 15616
Sep 29 01:03:54 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:03:54.099483 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:54.081411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.3.21.193:41483)
0929 01:03:54.082142 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:54.082152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:54.082187 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:03:54.082436 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:54.082446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:54.082735 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:54.082912 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:54.084376 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.085425 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.087230 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.087878 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.088038 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:54.092654 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:54.092670 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:54.092687 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:54.092727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:54.096255 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.097163 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.097176 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.097179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.097330 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.097725 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.097733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.097737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.098119 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:54.098441 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:54.098816 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 01:03:54.099134 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":309,"threads_started":1}
I20260929 01:03:54.101009 15744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:54.188988 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 15616
I20260929 01:03:54.206094 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 15755
Sep 29 01:03:54 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:03:54.233850 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:54.207256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.3.21.194:38277)
0929 01:03:54.208001 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:54.208009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:54.208036 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:03:54.208099 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:54.208105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:54.208336 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:54.208470 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:54.209912 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.211349 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.212971 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.213583 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.213699 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:54.227243 (+ 13544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:54.227261 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:54.227279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:54.227336 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:54.230881 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.231729 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.231737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.231739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.231917 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.232352 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.232356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.232358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.232725 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:54.232974 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:54.233275 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 01:03:54.233547 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":358,"threads_started":1}
I20260929 01:03:54.235262 15887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:54.322791 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 15755
I20260929 01:03:54.343430 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 15898
Sep 29 01:03:54 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:03:54.362437 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:54.344542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.3.21.195:38031)
0929 01:03:54.345221 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:54.345229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:54.345286 (+ 57us) server_negotiation.cc:407] Connection header received
0929 01:03:54.345357 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:54.345363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:54.345608 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:54.345744 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:54.347382 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.348539 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.350043 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:54.350797 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:54.350920 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:54.356068 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:54.356084 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:54.356101 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:54.356144 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:54.359291 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.360125 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.360132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.360135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.360274 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:54.360743 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:54.360747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:54.360749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:54.361142 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:54.361398 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:54.361738 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 01:03:54.362103 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":379,"threads_started":1}
I20260929 01:03:54.363978 16021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:54.452737 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 15898
I20260929 01:03:54.469000 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 15504
I20260929 01:03:54.470371 15566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:54.584390 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 15504
I20260929 01:03:54.610278 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:39169
--webserver_interface=127.3.21.254
--webserver_port=42421
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:39169
--ranger_config_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:54.982750 16038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:54.983088 16038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:54.983248 16038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:54.991227 16038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:03:54.991369 16038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:54.991458 16038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:54.991540 16038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:03:54.991608 16038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:03:55.004125 16038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:39169
--ranger_config_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:55.005579 16038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:55.007110 16038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:55.021670 16046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:55.021816 16043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:55.021765 16044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:55.022676 16038 server_base.cc:1034] running on GCE node
I20260929 01:03:55.023589 16038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:55.025516 16038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:55.026868 16038 hybrid_clock.cc:648] HybridClock initialized: now 1790643835026885 us; error 107 us; skew 500 ppm
I20260929 01:03:55.027454 16038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:55 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643835, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:55.031183 16038 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:03:55.033152 16038 webserver.cc:460] Webserver started at http://127.3.21.254:42421/ using document root <none> and password file <none>
I20260929 01:03:55.033900 16038 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:55.034051 16038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:03:55 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643835, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:55.102622 16038 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:03:55.107182 16054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:55.109298 16038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:03:55.109535 16038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "04add40b01aa4a4da33f527eb2bb2277"
format_stamp: "Formatted at 2026-09-29 01:03:50 on dist-test-slave-k0b7"
server_key: "5db342618cf4bb8e8961cff3325f58b1"
server_key_iv: "2980de9d2c008b52f6aeac255eac917b"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:55.110646 16038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:55.129379 16038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:55.134398 16038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:55.134809 16038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:55.156426 16038 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:39169
I20260929 01:03:55.156440 16105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:39169 every 8 connection(s)
I20260929 01:03:55.158568 16038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:03:55.165964 16106 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:03:55.168321 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16038
I20260929 01:03:55.169214 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:41483
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:39169
--webserver_port=40369
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:03:55.169891 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Bootstrap starting.
I20260929 01:03:55.189491 16106 log.cc:824] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:55.199608 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:03:55.200214 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Bootstrap complete.
I20260929 01:03:55.210160 16106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:55.211259 16106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04add40b01aa4a4da33f527eb2bb2277, State: Initialized, Role: FOLLOWER
I20260929 01:03:55.211907 16106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:55.212162 16106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:03:55.212297 16106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:03:55.212482 16106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:03:55.217111 16106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:55.217672 16106 leader_election.cc:302] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04add40b01aa4a4da33f527eb2bb2277; no voters:
I20260929 01:03:55.218415 16106 leader_election.cc:288] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:03:55.218585 16110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:03:55.219837 16110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [term 2 LEADER]: Becoming Leader. State: Replica: 04add40b01aa4a4da33f527eb2bb2277, State: Running, Role: LEADER
I20260929 01:03:55.220528 16110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } }
I20260929 01:03:55.220973 16106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:03:55.223527 16111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "04add40b01aa4a4da33f527eb2bb2277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } } }
I20260929 01:03:55.224027 16111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:55.228909 16112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04add40b01aa4a4da33f527eb2bb2277. Latest consensus state: current_term: 2 leader_uuid: "04add40b01aa4a4da33f527eb2bb2277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04add40b01aa4a4da33f527eb2bb2277" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 39169 } } }
I20260929 01:03:55.230046 16112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277 [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:55.231608 16116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:03:55.237870 16116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:03:55.239779 16116 catalog_manager.cc:1259] Loaded cluster ID: e6ee62ec5f4b4dc084523f78113b0b0a
I20260929 01:03:55.239991 16116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:03:55.244520 16116 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:03:55.247010 16116 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Loaded TSK: 0
I20260929 01:03:55.249221 16116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:03:55.254221 16106 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 01:03:55.642194 16108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:55.643060 16108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:55.643442 16108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:55.656177 16108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:55.656414 16108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:55.656700 16108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:03:55.671965 16108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:41483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=40369
--tserver_master_addrs=127.3.21.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:55.673478 16108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:55.675067 16108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:55.699458 16143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:55.701087 16144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:55.722194 16108 server_base.cc:1034] running on GCE node
W20260929 01:03:55.723413 16146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:55.724433 16108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:55.725922 16108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:55.727254 16108 hybrid_clock.cc:648] HybridClock initialized: now 1790643835727179 us; error 65 us; skew 500 ppm
I20260929 01:03:55.727711 16108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:55 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643835, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:55.732626 16108 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:03:55.734890 16108 webserver.cc:460] Webserver started at http://127.3.21.193:40369/ using document root <none> and password file <none>
I20260929 01:03:55.736088 16108 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:55.736330 16108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:03:55 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643835, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:55.779706 16108 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:03:55.790064 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:55.792285 16108 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.002s
I20260929 01:03:55.792600 16108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "e0698cdab2044a2089e58a836f3c10e9"
format_stamp: "Formatted at 2026-09-29 01:03:52 on dist-test-slave-k0b7"
server_key: "06a72e7c098380074475548be0a109a2"
server_key_iv: "9cd565b6f730094243de28c80c0024b7"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:55.794322 16108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:55.826402 16108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:55.832147 16108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:55.832540 16108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:55.838429 16108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:55.840751 16108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:55.841099 16108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:03:55.841322 16108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:55.841459 16108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:55.951110 16108 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:41483
I20260929 01:03:55.951193 16267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:41483 every 8 connection(s)
I20260929 01:03:55.953013 16108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:03:55.961419 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16108
I20260929 01:03:55.962473 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:38277
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:39169
--webserver_port=35483
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:03:56 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643835, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:03:56.028725 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:55.956641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39169 (local address 127.3.21.193:40953)
0929 01:03:55.995268 (+ 38627us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:55.995342 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:55.998171 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:56.000680 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:56.000694 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:56.001501 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:56.002547 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:56.002580 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.004756 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.004760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:56.006362 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:56.006382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.008443 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.008451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:56.008751 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:56.010265 (+ 1514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:56.010300 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:56.013426 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:56.022995 (+ 9569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:56.023009 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:56.023033 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:56.023863 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:56.024519 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:56.024524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:56.024528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:56.024793 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:56.025859 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:56.025868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:56.026297 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 01:03:56.026650 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 01:03:56.027276 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38237,"thread_start_us":38304,"threads_started":1}
I20260929 01:03:56.028825 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:55.959098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:40953 (local address 127.3.21.254:39169)
0929 01:03:55.995002 (+ 35904us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:55.995030 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:55.996102 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 01:03:55.999269 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:55.999290 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:55.999967 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:56.002477 (+ 2510us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:56.002953 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.004479 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.006619 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.007746 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.007973 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:56.013851 (+ 5878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:56.013943 (+ 92us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:56.013967 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:56.014063 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:56.022703 (+ 8640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:56.024112 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:56.024124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:56.024133 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:56.024382 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:56.024946 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:56.024951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:56.024954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:56.025691 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:56.025977 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:56.026958 (+ 981us) server_negotiation.cc:299] Negotiation successful
0929 01:03:56.027497 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35499,"thread_start_us":36907,"threads_started":1}
I20260929 01:03:56.032997 16268 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39169
I20260929 01:03:56.033334 16268 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:56.034200 16268 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
W20260929 01:03:56.037720 16268 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:39169 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:03:56.038009 16268 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:56.038314 16268 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:56.039724 16268 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:56.040109 16268 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
W20260929 01:03:56.041285 16268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:03:56.443127 16272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:56.443497 16272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:56.443675 16272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:56.456019 16272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:56.456182 16272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:56.456432 16272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:03:56.473192 16272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:38277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=35483
--tserver_master_addrs=127.3.21.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:56.475267 16272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:56.477219 16272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:56.525934 16288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:56.527420 16286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:56.538726 16291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:56.557101 16272 server_base.cc:1034] running on GCE node
I20260929 01:03:56.557997 16272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:56.559489 16272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:56.560777 16272 hybrid_clock.cc:648] HybridClock initialized: now 1790643836560682 us; error 76 us; skew 500 ppm
I20260929 01:03:56.561282 16272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:56 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643836, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:56.565721 16272 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:03:56.572098 16272 webserver.cc:460] Webserver started at http://127.3.21.194:35483/ using document root <none> and password file <none>
I20260929 01:03:56.572973 16272 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:56.573249 16272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:03:56 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643836, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:56.604317 16272 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:03:56.607863 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:56.609586 16272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:03:56.609800 16272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "5d87bbc936d44907911dbd84a961bb1b"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "996c0cdccc90f4bc6aa887fb5a3379d2"
server_key_iv: "77f44b532c302445f2e047ac2cac5696"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:56.611192 16272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 01:03:56 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643836, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:56.664599 16272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:56.670377 16272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:56.670920 16272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:56.672322 16272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:56.674147 16272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:56.674260 16272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:56.674366 16272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:56.674435 16272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:56.744915 16272 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:38277
I20260929 01:03:56.744988 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:38277 every 8 connection(s)
I20260929 01:03:56.747138 16272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:03:56.751910 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16272
I20260929 01:03:56.752926 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.21.212:44037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:38031
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:39169
--webserver_port=41767
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:03:56 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643836, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:03:56.796440 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:56.750958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39169 (local address 127.3.21.194:55283)
0929 01:03:56.762152 (+ 11194us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:56.762262 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:56.766896 (+ 4634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:56.772343 (+ 5447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:56.772359 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:56.773067 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:56.773968 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:56.773999 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.775751 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.775757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:56.777921 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:56.777950 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.779712 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.779723 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:56.780139 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:56.781116 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:56.781171 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:56.784817 (+ 3646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:56.790463 (+ 5646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:56.790477 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:56.790537 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:56.791405 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:56.792029 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:56.792037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:56.792043 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:56.792391 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:56.793314 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:56.793325 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:56.793854 (+ 529us) client_negotiation.cc:769] Sending connection context
0929 01:03:56.794211 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 01:03:56.794898 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10772,"thread_start_us":9517,"threads_started":1}
I20260929 01:03:56.797959 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:56.750294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:55283 (local address 127.3.21.254:39169)
0929 01:03:56.765442 (+ 15148us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:56.765449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:56.765592 (+ 143us) server_negotiation.cc:407] Connection header received
0929 01:03:56.768204 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:56.768212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:56.768315 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:56.768441 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:56.774241 (+ 5800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.775504 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.778156 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:56.779121 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:56.779299 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:56.785247 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:56.785258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:56.785268 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:56.785332 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:56.790164 (+ 4832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:56.791642 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:56.791652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:56.791657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:56.791851 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:56.792585 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:56.792594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:56.792600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:56.793075 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:56.793222 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:56.797116 (+ 3894us) server_negotiation.cc:299] Negotiation successful
0929 01:03:56.797410 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14639,"thread_start_us":14296,"threads_started":1}
I20260929 01:03:56.800604 16413 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39169
I20260929 01:03:56.801007 16413 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:56.802114 16413 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:56.804706 16071 ts_manager.cc:194] Registered new tserver with Master: 5d87bbc936d44907911dbd84a961bb1b (127.3.21.194:38277)
I20260929 01:03:56.807273 16071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:55283
I20260929 01:03:57.041492 16268 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:57.042039 16268 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:57.043365 16071 ts_manager.cc:194] Registered new tserver with Master: e0698cdab2044a2089e58a836f3c10e9 (127.3.21.193:41483)
I20260929 01:03:57.044533 16071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:40953
Sep 29 01:03:57 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643836, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
W20260929 01:03:57.250923 16417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:57.251307 16417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:57.251463 16417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:57.259337 16417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:57.259513 16417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:57.259714 16417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:03:57.271152 16417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.21.212:44037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:38031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=41767
--tserver_master_addrs=127.3.21.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:03:57.272596 16417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:57.273954 16417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:57.290637 16422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:57.291239 16423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:57.306962 16425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:57.307648 16417 server_base.cc:1034] running on GCE node
I20260929 01:03:57.308658 16417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:57.309952 16417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:57.311245 16417 hybrid_clock.cc:648] HybridClock initialized: now 1790643837311162 us; error 68 us; skew 500 ppm
I20260929 01:03:57.311685 16417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:03:57 dist-test-slave-k0b7 krb5kdc[15060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:57.316565 16417 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:03:57.320098 16417 webserver.cc:460] Webserver started at http://127.3.21.195:41767/ using document root <none> and password file <none>
I20260929 01:03:57.320986 16417 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:57.321257 16417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:03:57 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:57.352306 16417 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.003s
I20260929 01:03:57.363049 16433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:57.364959 16417 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.008s
I20260929 01:03:57.365186 16417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "4232363cd54b405abc58fb64de664bd2"
format_stamp: "Formatted at 2026-09-29 01:03:53 on dist-test-slave-k0b7"
server_key: "9eb6c2fb478eb76470b2285e46b203df"
server_key_iv: "951cdc7caedc996e3de942421deea840"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:57.366281 16417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:57.387900 16417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:57.393685 16417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:57.394047 16417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:57.395537 16417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:57.397256 16417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:57.397393 16417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:57.397554 16417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:57.397653 16417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:57.460173 16417 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:38031
I20260929 01:03:57.460237 16545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:38031 every 8 connection(s)
I20260929 01:03:57.462152 16417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:03:57.467056 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16417
I20260929 01:03:57.538715 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.468288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.3.21.254:39169)
0929 01:03:57.505274 (+ 36986us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.505280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.505309 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:03:57.505385 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.505390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.505497 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.505662 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:57.509262 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.510320 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.512663 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.513345 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.513494 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.518508 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:57.518516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:57.518521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:57.518575 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:57.531933 (+ 13358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:57.533270 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:57.533276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:57.533280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:57.533456 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:57.536884 (+ 3428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:57.536890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:57.536893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:57.537311 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:57.537615 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.538033 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.538316 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36834,"thread_start_us":33902,"threads_started":1}
Sep 29 01:03:57 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:03:57.551780 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.466579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:35443 (local address 127.3.21.254:39169)
0929 01:03:57.502160 (+ 35581us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.502170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.508149 (+ 5979us) server_negotiation.cc:407] Connection header received
0929 01:03:57.520702 (+ 12553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.520710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.520837 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.520918 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:57.532214 (+ 11296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.533507 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.536188 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.536880 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.536999 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.543832 (+ 6833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:57.543839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:57.543844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:57.543905 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:57.547080 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:57.548255 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:57.548262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:57.548265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:57.548522 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:57.549191 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:57.549197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:57.549201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:57.549615 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:57.549769 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.551064 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.551331 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35081,"thread_start_us":40304,"threads_started":1}
I20260929 01:03:57.553179 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.465846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:39169 (local address 127.3.21.195:35443)
0929 01:03:57.507738 (+ 41892us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:57.507818 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:57.520251 (+ 12433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:57.526399 (+ 6148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:57.526418 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:57.530800 (+ 4382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:57.531881 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.531918 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.533796 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.533802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.535977 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.535998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.537457 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.537468 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.537771 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.540466 (+ 2695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:57.540504 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:57.543489 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:57.547261 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:57.547273 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:57.547295 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:57.548070 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:57.548701 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:57.548706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:57.548710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:57.549036 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:57.549890 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:57.549899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:57.550436 (+ 537us) client_negotiation.cc:769] Sending connection context
0929 01:03:57.550908 (+ 472us) client_negotiation.cc:240] Negotiation successful
0929 01:03:57.551602 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41450,"thread_start_us":47667,"threads_started":1}
I20260929 01:03:57.555450 16546 heartbeater.cc:347] Connected to a master server at 127.3.21.254:39169
I20260929 01:03:57.555800 16546 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:57.556749 16546 heartbeater.cc:511] Master 127.3.21.254:39169 requested a full tablet report, sending...
I20260929 01:03:57.558887 16071 ts_manager.cc:194] Registered new tserver with Master: 4232363cd54b405abc58fb64de664bd2 (127.3.21.195:38031)
I20260929 01:03:57.560134 16071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:35443
I20260929 01:03:57.560551 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:03:57 dist-test-slave-k0b7 krb5kdc[15060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643836, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for HTTP/127.3.21.212@KRBTEST.COM
I20260929 01:03:57.601380 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.575960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.3.21.254:39169)
0929 01:03:57.577392 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.577402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.583158 (+ 5756us) server_negotiation.cc:407] Connection header received
0929 01:03:57.583374 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.583380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.583488 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.583605 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:57.585772 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.587088 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.589293 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.589946 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.590084 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.594002 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:57.594009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:57.594014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:57.594065 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:57.597708 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:57.598800 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:57.598807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:57.598811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:57.598998 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:57.599488 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:57.599493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:57.599496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:57.599908 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:57.600147 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.600757 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.601024 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231}
I20260929 01:03:57.609338 16071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:03:57.621209 16071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:03:57.671352 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.641994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:38277 (local address 127.0.0.1:44692)
0929 01:03:57.642981 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:57.643021 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:57.643204 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:57.644947 (+ 1743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:57.644956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:57.644994 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:57.646588 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.646611 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.662570 (+ 15959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.662579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.665220 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.665253 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.669935 (+ 4682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.669949 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.670362 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.670455 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 01:03:57.670638 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 01:03:57.670971 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":774,"threads_started":1}
I20260929 01:03:57.674856 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.642603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.3.21.194:38277)
0929 01:03:57.643752 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.643762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.643793 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:03:57.643899 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.643907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.644656 (+ 749us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.646892 (+ 2236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:57.647168 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.656546 (+ 9378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.665586 (+ 9040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.667307 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.673843 (+ 6536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.674109 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.674244 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.674458 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":2064,"threads_started":1}
I20260929 01:03:57.691474 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.646166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35432)
0929 01:03:57.672003 (+ 25837us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:57.672025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:57.672165 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:57.678979 (+ 6814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:57.678986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:57.679027 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:57.679248 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.679262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.685667 (+ 6405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.685676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.687955 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.687988 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.690623 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.690631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.690842 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.690893 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 01:03:57.690976 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:03:57.691143 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25721,"thread_start_us":30422,"threads_started":1}
I20260929 01:03:57.693078 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.646766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.3.21.193:41483)
0929 01:03:57.677245 (+ 30479us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.677256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.677292 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:03:57.677418 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.677436 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.677752 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.679480 (+ 1728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:57.679766 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.681383 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.688681 (+ 7298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.690356 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.692038 (+ 1682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.692315 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.692419 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.692712 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30259,"thread_start_us":36393,"threads_started":1}
I20260929 01:03:57.693214 16347 tablet_service.cc:1467] Processing CreateTablet for tablet 46f1f5bad6c640aba952c9d068c842f0 (DEFAULT_TABLE table=test-table [id=4f9bb5ca044248c48d50040e745ac5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:57.694981 16347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f1f5bad6c640aba952c9d068c842f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:57.699564 16203 tablet_service.cc:1467] Processing CreateTablet for tablet 46f1f5bad6c640aba952c9d068c842f0 (DEFAULT_TABLE table=test-table [id=4f9bb5ca044248c48d50040e745ac5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:57.701311 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f1f5bad6c640aba952c9d068c842f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:57.717265 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.686240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:38031 (local address 127.0.0.1:48218)
0929 01:03:57.688267 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:57.688300 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:57.688442 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:57.693790 (+ 5348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:57.693795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:57.693853 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:57.694265 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.694278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.710718 (+ 16440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.710725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.712882 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.712907 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.715697 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.715706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.715906 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.715953 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 01:03:57.716039 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:03:57.716208 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1869}
I20260929 01:03:57.718806 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.677443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.3.21.195:38031)
0929 01:03:57.693098 (+ 15655us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.693108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.693147 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:03:57.693256 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.693263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.693550 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.694506 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:57.694829 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.710454 (+ 15625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.713392 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.715350 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.717785 (+ 2435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.718085 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.718176 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.718378 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13725,"thread_start_us":15406,"threads_started":1}
I20260929 01:03:57.723814 16481 tablet_service.cc:1467] Processing CreateTablet for tablet 46f1f5bad6c640aba952c9d068c842f0 (DEFAULT_TABLE table=test-table [id=4f9bb5ca044248c48d50040e745ac5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:57.726222 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f1f5bad6c640aba952c9d068c842f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:57.746541 16568 tablet_bootstrap.cc:492] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Bootstrap starting.
I20260929 01:03:57.752331 16567 tablet_bootstrap.cc:492] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: Bootstrap starting.
I20260929 01:03:57.755568 16569 tablet_bootstrap.cc:492] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Bootstrap starting.
I20260929 01:03:57.758919 16567 tablet_bootstrap.cc:654] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:57.759732 16569 tablet_bootstrap.cc:654] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:57.760442 16567 log.cc:824] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:57.760891 16568 tablet_bootstrap.cc:654] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:57.761356 16569 log.cc:824] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:57.762184 16568 log.cc:824] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:57.775427 16567 tablet_bootstrap.cc:492] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: No bootstrap required, opened a new log
I20260929 01:03:57.775779 16567 ts_tablet_manager.cc:1397] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.005s
I20260929 01:03:57.781888 16568 tablet_bootstrap.cc:492] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: No bootstrap required, opened a new log
I20260929 01:03:57.782244 16568 ts_tablet_manager.cc:1397] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20260929 01:03:57.791285 16569 tablet_bootstrap.cc:492] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: No bootstrap required, opened a new log
I20260929 01:03:57.791731 16569 ts_tablet_manager.cc:1397] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.025s
I20260929 01:03:57.798154 16567 raft_consensus.cc:348] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.798815 16567 raft_consensus.cc:374] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:57.798924 16567 raft_consensus.cc:729] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0698cdab2044a2089e58a836f3c10e9, State: Initialized, Role: FOLLOWER
I20260929 01:03:57.799809 16567 consensus_queue.cc:260] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.804250 16569 raft_consensus.cc:348] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.804898 16569 raft_consensus.cc:374] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:57.805051 16569 raft_consensus.cc:729] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4232363cd54b405abc58fb64de664bd2, State: Initialized, Role: FOLLOWER
I20260929 01:03:57.805940 16569 consensus_queue.cc:260] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.807730 16568 raft_consensus.cc:348] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.808275 16568 raft_consensus.cc:374] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:57.808406 16568 raft_consensus.cc:729] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d87bbc936d44907911dbd84a961bb1b, State: Initialized, Role: FOLLOWER
I20260929 01:03:57.808594 16567 ts_tablet_manager.cc:1428] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: Time spent starting tablet: real 0.033s user 0.019s sys 0.002s
I20260929 01:03:57.809036 16268 heartbeater.cc:503] Master 127.3.21.254:39169 was elected leader, sending a full tablet report...
I20260929 01:03:57.812793 16413 heartbeater.cc:503] Master 127.3.21.254:39169 was elected leader, sending a full tablet report...
I20260929 01:03:57.816390 16569 ts_tablet_manager.cc:1428] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Time spent starting tablet: real 0.024s user 0.012s sys 0.008s
I20260929 01:03:57.817281 16546 heartbeater.cc:503] Master 127.3.21.254:39169 was elected leader, sending a full tablet report...
I20260929 01:03:57.827149 16568 consensus_queue.cc:260] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.836709 16568 ts_tablet_manager.cc:1428] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Time spent starting tablet: real 0.054s user 0.018s sys 0.003s
I20260929 01:03:57.839074 16575 raft_consensus.cc:482] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:03:57.839423 16575 raft_consensus.cc:504] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.841894 16575 leader_election.cc:288] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0698cdab2044a2089e58a836f3c10e9 (127.3.21.193:41483), 4232363cd54b405abc58fb64de664bd2 (127.3.21.195:38031)
I20260929 01:03:57.852955 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.842906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:57073 (local address 127.3.21.193:41483)
0929 01:03:57.843119 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.843126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.843780 (+ 654us) server_negotiation.cc:407] Connection header received
0929 01:03:57.844070 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.844076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.844177 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.844245 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:57.845372 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.846541 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.850161 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.851472 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.851644 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.851726 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.852496 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.852677 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:03:57.854015 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.842868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.3.21.194:57073)
0929 01:03:57.843657 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:57.843677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:57.843902 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:57.844824 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:57.844829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:57.844857 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:57.845185 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.845196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.847603 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.847609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.849925 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.849945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.851978 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.851986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.852248 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.852297 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 01:03:57.853435 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0929 01:03:57.853612 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":409,"threads_started":1}
I20260929 01:03:57.860728 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f1f5bad6c640aba952c9d068c842f0" candidate_uuid: "5d87bbc936d44907911dbd84a961bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0698cdab2044a2089e58a836f3c10e9" is_pre_election: true
I20260929 01:03:57.861477 16223 raft_consensus.cc:2454] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d87bbc936d44907911dbd84a961bb1b in term 0.
I20260929 01:03:57.862442 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.844514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:37661 (local address 127.3.21.195:38031)
0929 01:03:57.844701 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.844706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.854538 (+ 9832us) server_negotiation.cc:407] Connection header received
0929 01:03:57.854852 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.854858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.854937 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.855000 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:57.855676 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.856841 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.859627 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.860932 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.861086 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.861172 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.862010 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.862172 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:03:57.862352 16302 leader_election.cc:302] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d87bbc936d44907911dbd84a961bb1b, e0698cdab2044a2089e58a836f3c10e9; no voters:
I20260929 01:03:57.862812 16575 raft_consensus.cc:2781] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:03:57.862972 16575 raft_consensus.cc:482] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:03:57.863070 16575 raft_consensus.cc:3037] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:57.863512 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.843828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:38031 (local address 127.3.21.194:37661)
0929 01:03:57.854406 (+ 10578us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:57.854427 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:57.854668 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:57.855208 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:57.855214 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:57.855243 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:57.855497 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.855510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.857034 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.857039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.859029 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:57.859046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.861498 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.861507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:57.861796 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.861837 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:03:57.862862 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0929 01:03:57.863157 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10470,"thread_start_us":15486,"threads_started":1}
I20260929 01:03:57.864931 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f1f5bad6c640aba952c9d068c842f0" candidate_uuid: "5d87bbc936d44907911dbd84a961bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4232363cd54b405abc58fb64de664bd2" is_pre_election: true
I20260929 01:03:57.865546 16501 raft_consensus.cc:2454] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d87bbc936d44907911dbd84a961bb1b in term 0.
I20260929 01:03:57.866402 16575 raft_consensus.cc:504] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.867188 16575 leader_election.cc:288] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [CANDIDATE]: Term 1 election: Requested vote from peers e0698cdab2044a2089e58a836f3c10e9 (127.3.21.193:41483), 4232363cd54b405abc58fb64de664bd2 (127.3.21.195:38031)
I20260929 01:03:57.867980 16501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f1f5bad6c640aba952c9d068c842f0" candidate_uuid: "5d87bbc936d44907911dbd84a961bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4232363cd54b405abc58fb64de664bd2"
I20260929 01:03:57.868219 16501 raft_consensus.cc:3037] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:57.871217 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f1f5bad6c640aba952c9d068c842f0" candidate_uuid: "5d87bbc936d44907911dbd84a961bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0698cdab2044a2089e58a836f3c10e9"
I20260929 01:03:57.871482 16223 raft_consensus.cc:3037] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:57.875633 16501 raft_consensus.cc:2454] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d87bbc936d44907911dbd84a961bb1b in term 1.
I20260929 01:03:57.875728 16223 raft_consensus.cc:2454] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d87bbc936d44907911dbd84a961bb1b in term 1.
I20260929 01:03:57.876307 16302 leader_election.cc:302] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d87bbc936d44907911dbd84a961bb1b, e0698cdab2044a2089e58a836f3c10e9; no voters:
I20260929 01:03:57.876713 16575 raft_consensus.cc:2781] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:57.878019 16575 raft_consensus.cc:686] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 1 LEADER]: Becoming Leader. State: Replica: 5d87bbc936d44907911dbd84a961bb1b, State: Running, Role: LEADER
I20260929 01:03:57.878695 16575 consensus_queue.cc:237] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } }
I20260929 01:03:57.888681 16070 catalog_manager.cc:5495] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d87bbc936d44907911dbd84a961bb1b (127.3.21.194). New cstate: current_term: 1 leader_uuid: "5d87bbc936d44907911dbd84a961bb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d87bbc936d44907911dbd84a961bb1b" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20260929 01:03:57.947973 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:57.940462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.3.21.194:38277)
0929 01:03:57.940679 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:57.940685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:57.940707 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:03:57.941220 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:57.941225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:57.941326 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:57.941527 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:03:57.942315 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.943650 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.945215 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:57.945673 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:57.945789 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:57.946622 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:03:57.947282 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:03:57.947427 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:57.947649 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 01:03:57.947734 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:03:57.957650 16223 raft_consensus.cc:1261] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9 [term 1 FOLLOWER]: Refusing update from remote peer 5d87bbc936d44907911dbd84a961bb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:03:57.957873 16501 raft_consensus.cc:1261] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 1 FOLLOWER]: Refusing update from remote peer 5d87bbc936d44907911dbd84a961bb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:03:57.958905 16575 consensus_queue.cc:1035] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0698cdab2044a2089e58a836f3c10e9" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:03:57.959314 16578 consensus_queue.cc:1035] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4232363cd54b405abc58fb64de664bd2" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:03:57.973619 16588 mvcc.cc:205] Tried to move back new op lower bound from 7334477160261156864 to 7334477159956828160. Current Snapshot: MvccSnapshot[applied={T|T < 7334477160261156864}]
I20260929 01:03:57.988269 16071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:03:57.988663 16071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:03:57.993490 16071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04add40b01aa4a4da33f527eb2bb2277: Sending DeleteTablet for 3 replicas of tablet 46f1f5bad6c640aba952c9d068c842f0
I20260929 01:03:57.994987 16347 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f1f5bad6c640aba952c9d068c842f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:57 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:44692
I20260929 01:03:57.995023 16203 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f1f5bad6c640aba952c9d068c842f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:57 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:35432
I20260929 01:03:57.995214 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f1f5bad6c640aba952c9d068c842f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:57 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48218
I20260929 01:03:58.003165 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16108
I20260929 01:03:58.013543 16600 tablet_replica.cc:330] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: stopping tablet replica
I20260929 01:03:58.014106 16600 raft_consensus.cc:2229] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:03:58.014793 16600 raft_consensus.cc:2258] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:03:58.017369 16600 ts_tablet_manager.cc:1905] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:03:58.025352 16600 ts_tablet_manager.cc:1918] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:03:58.025559 16600 log.cc:1196] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/46f1f5bad6c640aba952c9d068c842f0
I20260929 01:03:58.026163 16600 ts_tablet_manager.cc:1939] T 46f1f5bad6c640aba952c9d068c842f0 P 4232363cd54b405abc58fb64de664bd2: Deleting consensus metadata
I20260929 01:03:58.027141 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.005368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.3.21.193:41483)
0929 01:03:58.005599 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:58.005608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:58.005634 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:03:58.005720 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:58.005728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:58.005839 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:58.005969 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:58.007976 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:58.009579 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:58.011684 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:58.012562 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:58.012722 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:58.017337 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:58.017368 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:58.017389 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:58.017448 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:58.023007 (+ 5559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:58.024291 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:58.024305 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:58.024309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:58.024507 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:58.024995 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:58.025002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:58.025005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:58.025659 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:58.026024 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:58.026430 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 01:03:58.026757 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:03:58.027928 16263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:58.028375 16057 catalog_manager.cc:4841] TS 4232363cd54b405abc58fb64de664bd2 (127.3.21.195:38031): tablet 46f1f5bad6c640aba952c9d068c842f0 (table test-table [id=4f9bb5ca044248c48d50040e745ac5c2]) successfully deleted
I20260929 01:03:58.034646 16599 tablet_replica.cc:330] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: stopping tablet replica
I20260929 01:03:58.035840 16599 raft_consensus.cc:2229] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:03:58.036819 16599 raft_consensus.cc:2258] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:03:58.038884 16599 ts_tablet_manager.cc:1905] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:03:58.044723 16599 ts_tablet_manager.cc:1918] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:03:58.044970 16599 log.cc:1196] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/46f1f5bad6c640aba952c9d068c842f0
I20260929 01:03:58.045506 16599 ts_tablet_manager.cc:1939] T 46f1f5bad6c640aba952c9d068c842f0 P 5d87bbc936d44907911dbd84a961bb1b: Deleting consensus metadata
I20260929 01:03:58.047422 16057 catalog_manager.cc:4841] TS 5d87bbc936d44907911dbd84a961bb1b (127.3.21.194:38277): tablet 46f1f5bad6c640aba952c9d068c842f0 (table test-table [id=4f9bb5ca044248c48d50040e745ac5c2]) successfully deleted
I20260929 01:03:58.176926 16598 tablet_replica.cc:330] T 46f1f5bad6c640aba952c9d068c842f0 P e0698cdab2044a2089e58a836f3c10e9: stopping tablet replica
I20260929 01:03:58.177295 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16108
W20260929 01:03:58.194799 16057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.21.193:41483 (error 108)
I20260929 01:03:58.194981 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16272
I20260929 01:03:58.196995 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.196449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35446)
0929 01:03:58.196675 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.196798 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 01:03:58.197355 16057 catalog_manager.cc:4584] TS e0698cdab2044a2089e58a836f3c10e9 (127.3.21.193:41483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46f1f5bad6c640aba952c9d068c842f0: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
I20260929 01:03:58.213896 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.195910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.3.21.194:38277)
0929 01:03:58.196092 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:58.196101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:58.196193 (+ 92us) server_negotiation.cc:407] Connection header received
0929 01:03:58.196417 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:58.196422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:58.196523 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:58.196636 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:58.199121 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:58.200207 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:58.202222 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:58.202817 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:58.202935 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:58.206807 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:58.206827 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:58.206846 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:58.206898 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:58.210338 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:58.211424 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:58.211432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:58.211434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:58.211612 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:58.212053 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:58.212057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:58.212059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:58.212553 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:58.212839 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:58.213243 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 01:03:58.213482 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:03:58.214774 16408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:58.251523 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.250916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35448)
0929 01:03:58.251184 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.251322 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 01:03:58.252861 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.252453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35460)
0929 01:03:58.252631 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.252720 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 01:03:58.313627 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.312989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35470)
0929 01:03:58.313294 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.313408 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 01:03:58.314908 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.314459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35484)
0929 01:03:58.314643 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.314733 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 01:03:58.331405 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16272
I20260929 01:03:58.350579 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16417
I20260929 01:03:58.367080 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.351597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.3.21.195:38031)
0929 01:03:58.351816 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:58.351821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:58.351842 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:03:58.352146 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:58.352151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:58.352228 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:58.352344 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:58.353854 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:58.354953 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:58.356561 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:58.357105 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:58.357219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:58.360337 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:58.360356 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:58.360372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:58.360419 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:58.363550 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:58.364360 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:58.364368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:58.364370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:58.364507 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:58.365005 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:58.365009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:58.365012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:58.365459 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:58.365747 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:58.366314 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 01:03:58.366636 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 01:03:58.368000 16541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:58.396344 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.395732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35500)
0929 01:03:58.395999 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.396111 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 01:03:58.397626 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.397172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35516)
0929 01:03:58.397399 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.397488 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 01:03:58.462702 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16417
I20260929 01:03:58.480924 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16038
I20260929 01:03:58.482339 16101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:03:58.602694 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:58.601970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41483 (local address 127.0.0.1:35520)
0929 01:03:58.602364 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:58.602474 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 01:03:58.602701 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16038
I20260929 01:03:58.625198 3159 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 01:03:58.962307 3159 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 01:03:58.962590 3159 mini_ranger.cc:66] Stopping Ranger...
I20260929 01:03:59.320436 3159 mini_ranger.cc:68] Stopped Ranger
2026-09-29 01:03:59.320 UTC [15090] LOG: received smart shutdown request
2026-09-29 01:03:59.324 UTC [15090] LOG: background worker "logical replication launcher" (PID 15099) exited with exit code 1
2026-09-29 01:03:59.325 UTC [15094] LOG: shutting down
2026-09-29 01:03:59.574 UTC [15090] LOG: database system is shut down
2026-09-29T01:03:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (61179 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:03:59 dist-test-slave-k0b7 krb5kdc[16614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:03:59 dist-test-slave-k0b7 krb5kdc[16614](info): set up 2 sockets
Sep 29 01:03:59 dist-test-slave-k0b7 krb5kdc[16614](info): commencing operation
krb5kdc: starting...
W20260929 01:04:00.866984 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:04:01 dist-test-slave-k0b7 krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:04:01.131847 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:41231
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:41231,127.3.21.253:34941,127.3.21.252:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:01.505893 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:01.506220 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:01.506397 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:01.514079 16627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:01.514251 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:01.514364 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:01.514559 16627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:01.514617 16627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:01.525475 16627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:41231,127.3.21.253:34941,127.3.21.252:44367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:01.527107 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:01.528589 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:01.542690 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:01.542719 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:01.542788 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:01.543540 16627 server_base.cc:1034] running on GCE node
I20260929 01:04:01.544396 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:01.546198 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:01.547456 16627 hybrid_clock.cc:648] HybridClock initialized: now 1790643841547391 us; error 96 us; skew 500 ppm
I20260929 01:04:01.547883 16627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:01 dist-test-slave-k0b7 krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:01.551424 16627 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:01.553694 16627 webserver.cc:460] Webserver started at http://127.3.21.254:41343/ using document root <none> and password file <none>
I20260929 01:04:01.554447 16627 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:01.554648 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:01.554935 16627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:01.558256 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "c5c021558e4746e48dd0efbbd19f6185"
format_stamp: "Formatted at 2026-09-29 01:04:01 on dist-test-slave-k0b7"
server_key: "2d19acae50e023fd2f9f156163ebb7d3"
server_key_iv: "b195c468eb4b64b7b285fb859626ebbc"
server_key_version: "encryptionkey@0"
I20260929 01:04:01.559190 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "c5c021558e4746e48dd0efbbd19f6185"
format_stamp: "Formatted at 2026-09-29 01:04:01 on dist-test-slave-k0b7"
server_key: "2d19acae50e023fd2f9f156163ebb7d3"
server_key_iv: "b195c468eb4b64b7b285fb859626ebbc"
server_key_version: "encryptionkey@0"
I20260929 01:04:01.565023 16627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:04:01.569290 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:01.571034 16627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:01.571264 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "c5c021558e4746e48dd0efbbd19f6185"
format_stamp: "Formatted at 2026-09-29 01:04:01 on dist-test-slave-k0b7"
server_key: "2d19acae50e023fd2f9f156163ebb7d3"
server_key_iv: "b195c468eb4b64b7b285fb859626ebbc"
server_key_version: "encryptionkey@0"
I20260929 01:04:01.571702 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:01.587028 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:01.592581 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:01.592948 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:01.614614 16627 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:41231
I20260929 01:04:01.614674 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:41231 every 8 connection(s)
I20260929 01:04:01.616165 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:01.618206 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16627
I20260929 01:04:01.618541 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:04:01.619427 3159 external_mini_cluster.cc:1428] Setting key 073386847aca09d705b53f4b49c19df9
I20260929 01:04:01.620550 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:01.627789 16694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.21.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.253@KRBTEST.COM" created.
Sep 29 01:04:01 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:01.654134 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.631657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41231 (local address 127.0.0.1:54512)
0929 01:04:01.633015 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.633057 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:01.635043 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:01.635973 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:01.635980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:01.636764 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:01.637104 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:01.637122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:01.638943 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:01.638951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:01.640229 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:01.640242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:01.641229 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:01.641239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:01.641401 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:01.642161 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:01.642207 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:01.644912 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:01.648721 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:01.648738 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:01.648755 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:01.649509 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:01.650055 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:01.650063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:01.650068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:01.650343 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:01.651434 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:01.651446 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:01.651726 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 01:04:01.652033 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 01:04:01.652744 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":113,"spinlock_wait_cycles":18816,"thread_start_us":898,"threads_started":1}
I20260929 01:04:01.654105 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.631657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.3.21.254:41231)
0929 01:04:01.632689 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:01.632707 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:01.633206 (+ 499us) server_negotiation.cc:407] Connection header received
0929 01:04:01.635470 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:01.635476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:01.635742 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:01.635961 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:01.637322 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:01.638776 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:01.640375 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:01.640979 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:01.641324 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:01.645192 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:01.645201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:01.645215 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:01.645245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:01.648479 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:01.649674 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:01.649686 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:01.649692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:01.649899 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:01.650618 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:01.650623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:01.650625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:01.651262 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:01.651494 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:01.652201 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 01:04:01.652500 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":494,"threads_started":1}
I20260929 01:04:01.657907 16694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:01.659349 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.658807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44430)
0929 01:04:01.659030 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.659176 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 01:04:01.659682 16646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
I20260929 01:04:01.661440 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.661038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44432)
0929 01:04:01.661221 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.661292 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:04:01.661664 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:34941: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab.
Entry for principal kudu/127.3.21.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab.
I20260929 01:04:01.679556 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:34941
--webserver_interface=127.3.21.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab
--principal=kudu/127.3.21.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:41231,127.3.21.253:34941,127.3.21.252:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:01.711050 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } attempt: 1
I20260929 01:04:01.712388 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.711822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44442)
0929 01:04:01.712070 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.712215 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 01:04:01.713626 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.713202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44450)
0929 01:04:01.713372 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.713463 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 01:04:01.713881 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:34941: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
I20260929 01:04:01.782204 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } attempt: 2
I20260929 01:04:01.784017 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.783255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44452)
0929 01:04:01.783552 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.783696 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 01:04:01.785517 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.785023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44458)
0929 01:04:01.785226 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.785327 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 01:04:01.785794 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:34941: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
I20260929 01:04:01.877164 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } attempt: 3
I20260929 01:04:01.878850 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.878245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44474)
0929 01:04:01.878551 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.878665 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 01:04:01.880442 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:01.879738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44484)
0929 01:04:01.880069 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:01.880200 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260929 01:04:01.880708 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:34941: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
I20260929 01:04:02.024041 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } attempt: 4
I20260929 01:04:02.025678 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.025021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44492)
0929 01:04:02.025402 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.025530 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260929 01:04:02.027369 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.026806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44496)
0929 01:04:02.027099 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.027207 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 01:04:02.027654 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:34941: Network error: Client connection negotiation failed: client connection to 127.3.21.253:34941: connect: Connection refused (error 111)
W20260929 01:04:02.041507 16701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:02.041836 16701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:02.042001 16701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:02.050314 16701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:02.050459 16701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:02.050593 16701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:02.050680 16701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:02.050768 16701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:02.062312 16701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:41231,127.3.21.253:34941,127.3.21.252:44367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab
--principal=kudu/127.3.21.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:34941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.21.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:02.063822 16701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:02.065192 16701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:02.079054 16706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:02.079088 16707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:02.079209 16709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:02.080217 16701 server_base.cc:1034] running on GCE node
I20260929 01:04:02.081233 16701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:02.082804 16701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:02.084069 16701 hybrid_clock.cc:648] HybridClock initialized: now 1790643842084004 us; error 66 us; skew 500 ppm
I20260929 01:04:02.084462 16701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:02.088486 16701 init.cc:377] Logged in from keytab as kudu/127.3.21.253@KRBTEST.COM (short username kudu)
I20260929 01:04:02.090175 16701 webserver.cc:460] Webserver started at http://127.3.21.253:38031/ using document root <none> and password file <none>
I20260929 01:04:02.091054 16701 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:02.091171 16701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:02.091429 16701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:02.094712 16701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data/instance:
uuid: "277622c0c7cd4ee4975767f722182356"
format_stamp: "Formatted at 2026-09-29 01:04:02 on dist-test-slave-k0b7"
server_key: "7b04de6cb239566fed7d9dc3a42160a0"
server_key_iv: "7f5e3cdad2277f35f4bc237679a024b4"
server_key_version: "encryptionkey@0"
I20260929 01:04:02.095695 16701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal/instance:
uuid: "277622c0c7cd4ee4975767f722182356"
format_stamp: "Formatted at 2026-09-29 01:04:02 on dist-test-slave-k0b7"
server_key: "7b04de6cb239566fed7d9dc3a42160a0"
server_key_iv: "7f5e3cdad2277f35f4bc237679a024b4"
server_key_version: "encryptionkey@0"
I20260929 01:04:02.101799 16701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260929 01:04:02.106271 16716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:02.108111 16701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:04:02.108299 16701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal
uuid: "277622c0c7cd4ee4975767f722182356"
format_stamp: "Formatted at 2026-09-29 01:04:02 on dist-test-slave-k0b7"
server_key: "7b04de6cb239566fed7d9dc3a42160a0"
server_key_iv: "7f5e3cdad2277f35f4bc237679a024b4"
server_key_version: "encryptionkey@0"
I20260929 01:04:02.108533 16701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:02.132359 16701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:02.137573 16701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:02.137843 16701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:02.158746 16701 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.253:34941
I20260929 01:04:02.158787 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.253:34941 every 8 connection(s)
I20260929 01:04:02.160171 16701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
I20260929 01:04:02.164345 16768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:02.165779 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16701
I20260929 01:04:02.165995 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-1/wal/instance
I20260929 01:04:02.166594 3159 external_mini_cluster.cc:1428] Setting key 512ef44698137c45c757b7e98e0b4a8a
I20260929 01:04:02.175227 16768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.21.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.252@KRBTEST.COM" created.
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:02.202591 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.178695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.3.21.254:41231)
0929 01:04:02.179530 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.179535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.180727 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 01:04:02.182893 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.182899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.183009 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.183133 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.184889 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.185681 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.187450 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.187914 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.188060 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.195817 (+ 7757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.195823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.195826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.195874 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.198985 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.199939 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.199944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.199947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.200081 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.200582 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.200586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.200588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.200913 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.201077 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.201887 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.202140 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":478,"threads_started":1}
I20260929 01:04:02.203629 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.179453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41231 (local address 127.0.0.1:54518)
0929 01:04:02.180529 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.180580 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.182583 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.183417 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.183426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.184026 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.184684 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.184703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.185873 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.185877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.187281 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.187295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.188175 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.188182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.188445 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.189304 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.189329 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.195470 (+ 6141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.199210 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.199221 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.199252 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.199767 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.200207 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.200212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.200216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.200444 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.201056 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.201066 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.201491 (+ 425us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.201780 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.202356 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":577,"threads_started":1}
I20260929 01:04:02.205716 16768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab.
Entry for principal kudu/127.3.21.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab.
I20260929 01:04:02.227679 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:44367
--webserver_interface=127.3.21.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab
--principal=kudu/127.3.21.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:41231,127.3.21.253:34941,127.3.21.252:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:02.227934 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.206717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44512)
0929 01:04:02.207068 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.207089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.207219 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.208513 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.208521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.209155 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.209403 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.209420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.211430 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.211438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.213185 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.213209 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.214736 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.214750 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.214995 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.216114 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.216133 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.219217 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.223653 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.223658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.223661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.224445 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.224942 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.224946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.224948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.225153 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.226335 (+ 1182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.226345 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.226905 (+ 560us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.227261 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.227545 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:04:02.228643 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.207066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.3.21.253:34941)
0929 01:04:02.208017 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.208024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.208052 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:04:02.208102 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.208108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.208312 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.208444 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.209980 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.211248 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.213457 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.214454 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.214642 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.219526 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.219549 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.219631 (+ 82us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.219725 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.223432 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.224628 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.224636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.224638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.224781 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.225280 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.225284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.225286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.225810 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.226193 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.227481 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.227809 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":640,"threads_started":1}
I20260929 01:04:02.231652 16768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:02.232986 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.232428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37186)
0929 01:04:02.232669 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.232790 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 01:04:02.233379 16717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.235481 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.234928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37196)
0929 01:04:02.235137 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.235280 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 01:04:02.235828 16768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.285354 16768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 1
I20260929 01:04:02.286803 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.286242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37200)
0929 01:04:02.286515 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.286642 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 01:04:02.287930 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.287564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37216)
0929 01:04:02.287719 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.287815 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:04:02.288276 16768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.326984 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } attempt: 5
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:04:02.344760 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.327907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44522)
0929 01:04:02.328154 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.328179 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.328403 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.328940 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.328944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.329387 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.329579 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.329589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.331124 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.331129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.332451 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.332492 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.333602 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.333607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.333753 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.334902 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.334925 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.337336 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.341632 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.341659 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.341662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.342274 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.342833 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.342837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.342840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.343065 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.343860 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.343863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.344092 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.344250 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.344500 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 01:04:02.344987 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.328186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.3.21.253:34941)
0929 01:04:02.328408 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.328414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.328439 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:04:02.328537 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.328543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.328652 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.328772 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.329771 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.330924 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.332695 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.333399 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.333559 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.337627 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.337635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.337640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.337688 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.341372 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.342444 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.342452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.342458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.342677 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.343216 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.343223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.343229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.343696 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.343883 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.344390 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.344656 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 01:04:02.346095 16694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:02.348073 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.347097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37228)
0929 01:04:02.347673 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.347769 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260929 01:04:02.349239 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.348820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37244)
0929 01:04:02.348991 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.349084 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 01:04:02.349459 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.356555 16768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 2
I20260929 01:04:02.358234 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.357484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37254)
0929 01:04:02.357816 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.358058 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260929 01:04:02.359449 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.359071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37262)
0929 01:04:02.359239 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.359318 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:04:02.359679 16768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.400739 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 1
I20260929 01:04:02.402347 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.401704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37270)
0929 01:04:02.402042 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.402156 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 01:04:02.403472 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.403089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37272)
0929 01:04:02.403249 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.403344 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 01:04:02.403714 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.450918 16768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 3
I20260929 01:04:02.452296 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.451690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37276)
0929 01:04:02.452040 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.452132 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 01:04:02.453377 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.452942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37284)
0929 01:04:02.453178 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.453254 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 01:04:02.453596 16768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.471971 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 2
I20260929 01:04:02.473316 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.472806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37290)
0929 01:04:02.473057 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.473145 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 01:04:02.474390 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.474027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37306)
0929 01:04:02.474184 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.474263 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:04:02.474634 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.580961 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 3
I20260929 01:04:02.582261 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.581686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37310)
0929 01:04:02.581924 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.582012 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 01:04:02.583595 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.583180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37312)
0929 01:04:02.583393 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.583481 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 01:04:02.583871 16694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
I20260929 01:04:02.596807 16768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 4
I20260929 01:04:02.598076 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.597555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37328)
0929 01:04:02.597810 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.597909 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 01:04:02.599205 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.598821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37342)
0929 01:04:02.599009 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.599092 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:04:02.599427 16768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:44367: Network error: Client connection negotiation failed: client connection to 127.3.21.252:44367: connect: Connection refused (error 111)
W20260929 01:04:02.612215 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:02.612592 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:02.612761 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:02.620860 16776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:02.621024 16776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:02.621138 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:02.621243 16776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:02.621325 16776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:02.632167 16776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:41231,127.3.21.253:34941,127.3.21.252:44367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab
--principal=kudu/127.3.21.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.21.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:02.633638 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:02.635108 16776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:02.648262 16781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:02.648209 16782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:02.648264 16784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:02.649427 16776 server_base.cc:1034] running on GCE node
I20260929 01:04:02.650234 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:02.651793 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:02.653021 16776 hybrid_clock.cc:648] HybridClock initialized: now 1790643842652974 us; error 35 us; skew 500 ppm
I20260929 01:04:02.653360 16776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:02.656888 16776 init.cc:377] Logged in from keytab as kudu/127.3.21.252@KRBTEST.COM (short username kudu)
I20260929 01:04:02.658811 16776 webserver.cc:460] Webserver started at http://127.3.21.252:35959/ using document root <none> and password file <none>
I20260929 01:04:02.659646 16776 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:02.659793 16776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:02.660166 16776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:02.663841 16776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data/instance:
uuid: "a3e92f814f76422f8418bc04777d10cd"
format_stamp: "Formatted at 2026-09-29 01:04:02 on dist-test-slave-k0b7"
server_key: "fe86218687a9b6ea260fa5cee6e62a92"
server_key_iv: "801db07a7cf0d783325229d6bfe502b2"
server_key_version: "encryptionkey@0"
I20260929 01:04:02.664870 16776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal/instance:
uuid: "a3e92f814f76422f8418bc04777d10cd"
format_stamp: "Formatted at 2026-09-29 01:04:02 on dist-test-slave-k0b7"
server_key: "fe86218687a9b6ea260fa5cee6e62a92"
server_key_iv: "801db07a7cf0d783325229d6bfe502b2"
server_key_version: "encryptionkey@0"
I20260929 01:04:02.672417 16776 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 01:04:02.677345 16791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:02.679234 16776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:04:02.679418 16776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal
uuid: "a3e92f814f76422f8418bc04777d10cd"
format_stamp: "Formatted at 2026-09-29 01:04:02 on dist-test-slave-k0b7"
server_key: "fe86218687a9b6ea260fa5cee6e62a92"
server_key_iv: "801db07a7cf0d783325229d6bfe502b2"
server_key_version: "encryptionkey@0"
I20260929 01:04:02.679634 16776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:02.701700 16776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:02.707417 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:02.707700 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:02.729074 16776 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.252:44367
I20260929 01:04:02.729274 16842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.252:44367 every 8 connection(s)
I20260929 01:04:02.730660 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
I20260929 01:04:02.734687 16843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:02.736590 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16776
I20260929 01:04:02.736794 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643611838815-3159-0/minicluster-data/master-2/wal/instance
I20260929 01:04:02.737447 3159 external_mini_cluster.cc:1428] Setting key d4ac0bacad839cc00c258fe4cccc00b8
I20260929 01:04:02.745218 16843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:02.761138 16694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 4
I20260929 01:04:02.763665 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.739590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.3.21.252:44367)
0929 01:04:02.740463 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.740476 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.740527 (+ 51us) server_negotiation.cc:407] Connection header received
0929 01:04:02.742194 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.742220 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.742775 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.743123 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.745129 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.746797 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.748751 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.749865 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.750073 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.754469 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.754539 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.754559 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.754620 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.758040 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.759005 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.759018 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.759031 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.759252 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.759711 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.759718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.759722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.760550 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.760890 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.761355 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.761895 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":432,"threads_started":1}
I20260929 01:04:02.766522 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.748435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:41231 (local address 127.0.0.1:54532)
0929 01:04:02.749371 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.749415 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.749591 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.750241 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.750274 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.751046 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.751352 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.751362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.752394 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.752404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.753547 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.753564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.754407 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.754418 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.754643 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.755352 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.755390 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.757909 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.762573 (+ 4664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.762606 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.762613 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.763193 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.763772 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.763779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.763786 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.764030 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.764952 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.764962 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.765203 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.765449 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.765866 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":498,"threads_started":1}
I20260929 01:04:02.767634 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.748374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.3.21.254:41231)
0929 01:04:02.748998 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.749005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.749532 (+ 527us) server_negotiation.cc:407] Connection header received
0929 01:04:02.749829 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.749834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.750006 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.750165 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.751523 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.752219 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.753701 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.754128 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.754266 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.758139 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.758145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.758148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.758184 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.762371 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.763384 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.763389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.763392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.763555 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.764202 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.764212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.764217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.764686 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.764862 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.766887 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.767144 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
I20260929 01:04:02.770267 16843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:02.784801 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.761881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37370)
0929 01:04:02.762146 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.762166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.762301 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.768170 (+ 5869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.768178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.768711 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.768953 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.768967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.770690 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.770696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.772239 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.772257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.773319 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.773326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.773510 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.774276 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.774295 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.776967 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.781389 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.781395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.781400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.782023 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.782513 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.782518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.782522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.782734 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.783439 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.783448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.783671 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.783844 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.784273 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1}
I20260929 01:04:02.785763 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.767025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.3.21.252:44367)
0929 01:04:02.767803 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.767810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.767827 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:04:02.767882 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.767888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.767964 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.769116 (+ 1152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.769440 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.770532 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.772438 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.773139 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.773272 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.777595 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.777602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.777607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.777636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.781233 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.782156 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.782163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.782170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.782336 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.782895 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.782899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.782901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.783276 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.783483 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.785320 (+ 1837us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.785523 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651}
I20260929 01:04:02.788902 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.771367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.3.21.253:34941)
0929 01:04:02.771561 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.771569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.771591 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:04:02.771657 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.771663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.771736 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.771846 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.772785 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.773705 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.775587 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.776338 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.776508 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.780858 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.780866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.780871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.780918 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.784703 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.786232 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.786238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.786242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.786453 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.787137 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.787144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.787148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.787454 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.787638 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.788213 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.788485 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:04:02.789470 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.768953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.3.21.254:41231)
0929 01:04:02.769193 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.769199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.770816 (+ 1617us) server_negotiation.cc:407] Connection header received
0929 01:04:02.770993 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.770999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.771106 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.771222 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.772899 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.773814 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.775769 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.776201 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.776349 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.782065 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.782073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.782080 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.782121 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.785942 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.786946 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.786952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.786955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.787119 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.787583 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.787589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.787592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.787967 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.788277 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.788623 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.788888 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":1259,"threads_started":1}
I20260929 01:04:02.790293 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.771179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:34941 (local address 127.0.0.1:44526)
0929 01:04:02.771454 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.771473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.771568 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.771902 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.771907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.772407 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.772620 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.772629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.773914 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.773921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.775404 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.775422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.776568 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.776574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.776794 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.777576 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.777599 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.780610 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.785173 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.785183 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.785187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.786096 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.786603 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.786610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.786617 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.786822 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.787708 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.787713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.787893 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.789399 (+ 1506us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.789922 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 01:04:02.791702 16843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:02.792279 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: Bootstrap starting.
I20260929 01:04:02.797040 16694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:02.798460 16694 log.cc:824] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:04:02.802028 16694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: No bootstrap required, opened a new log
I20260929 01:04:02.807981 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.792598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37384)
0929 01:04:02.793096 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.793114 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.793212 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.793756 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.793761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.794313 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.794577 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.794592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.795668 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.795676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.796630 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.796670 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.797732 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.797740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.797997 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.798585 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.798609 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.801499 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.805044 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.805050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.805054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.805603 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.805999 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.806004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.806007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.806137 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.806749 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.806754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.806901 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.807042 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.807412 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":4}
I20260929 01:04:02.808007 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.792946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.3.21.252:44367)
0929 01:04:02.793403 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.793408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.793422 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:04:02.793460 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.793464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.793524 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.793605 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.794741 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.795528 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.796824 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.797549 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.797696 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.801718 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.801722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.801725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.801757 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.804888 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.805740 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.805747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.805750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.805874 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.806237 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.806244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.806248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.806635 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.806787 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.807145 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.807575 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":349}
I20260929 01:04:02.811234 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.791669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.3.21.253:34941)
0929 01:04:02.791853 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.791860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.792011 (+ 151us) server_negotiation.cc:407] Connection header received
0929 01:04:02.792213 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.792219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.792300 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.792466 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.793855 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.794730 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.796507 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.797095 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.797238 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.803451 (+ 6213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.803461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.803467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.803513 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.807156 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.808145 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.808153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.808156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.808318 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.808763 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.808770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.808774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.809132 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.809423 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.810610 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.810868 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:04:02.813258 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: Bootstrap starting.
I20260929 01:04:02.813316 16694 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.813771 16694 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:02.813905 16694 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c021558e4746e48dd0efbbd19f6185, State: Initialized, Role: FOLLOWER
I20260929 01:04:02.814735 16694 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.816632 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:02.817165 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:02.817544 16694 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:02.818127 16843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:02.818277 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:04:02.819557 16843 log.cc:824] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:02.823138 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: No bootstrap required, opened a new log
I20260929 01:04:02.833448 16843 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.834064 16843 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:02.834203 16843 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e92f814f76422f8418bc04777d10cd, State: Initialized, Role: FOLLOWER
I20260929 01:04:02.835265 16843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.842424 16870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:02.842908 16870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:02.844863 16843 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:04:02.845923 16867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:04:02.846185 16867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:04:02.846639 16648 tablet.cc:2369] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:02.834822 16627 master_runner.cc:186] Error getting master registration for 127.3.21.253:34941: OK, instance_id { permanent_uuid: "277622c0c7cd4ee4975767f722182356" instance_seqno: 1790643842152865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:04:02.849517 16627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:04:02.869571 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.829831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.3.21.254:41231)
0929 01:04:02.830114 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.830121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.848818 (+ 18697us) server_negotiation.cc:407] Connection header received
0929 01:04:02.849173 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.849183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.849325 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.849435 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.852585 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.854053 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.856434 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.857208 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.857388 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.861490 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.861498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.861503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.861593 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.865695 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.866760 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.866767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.866770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.866955 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.867465 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.867470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.867474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.867912 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.868213 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.868855 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.869137 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 01:04:02.882871 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.836516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.3.21.253:34941)
0929 01:04:02.839550 (+ 3034us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.839557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.854886 (+ 15329us) server_negotiation.cc:407] Connection header received
0929 01:04:02.855430 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.855437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.855528 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.855598 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.858961 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.860144 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.864853 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.865584 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.865694 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.874245 (+ 8551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.874256 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.874262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.874311 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.878888 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.880000 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.880006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.880010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.880181 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.880726 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.880732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.880736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.881156 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.881425 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.882118 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.882429 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2874}
I20260929 01:04:02.885578 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.847960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.3.21.252:44367)
0929 01:04:02.848139 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.848148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.858665 (+ 10517us) server_negotiation.cc:407] Connection header received
0929 01:04:02.862763 (+ 4098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.862769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.862883 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.862964 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.866807 (+ 3843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.867976 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.872690 (+ 4714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.873191 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.873280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.877116 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.877123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.877128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.877199 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.881030 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.882118 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.882125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.882129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.882283 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.882763 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.882768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.882771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.883299 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.883554 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.885023 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.885286 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:04:02.898638 16768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } attempt: 5
Sep 29 01:04:02 dist-test-slave-k0b7 krb5kdc[16614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
W20260929 01:04:02.924081 16883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:04:02.924216 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.899267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:44367 (local address 127.0.0.1:37388)
0929 01:04:02.899531 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:02.899553 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:02.899675 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:02.900265 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:02.900274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:02.900876 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:02.901099 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.901112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.902920 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.902930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.904522 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:02.904540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.906040 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.906046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:02.906353 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.907185 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:02.907211 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:02.910331 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:02.919743 (+ 9412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.919749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.919753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.920394 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.921779 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:02.921785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:02.921790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:02.921981 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:02.923003 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:02.923008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:02.923202 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 01:04:02.923400 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 01:04:02.923761 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
W20260929 01:04:02.924795 16883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:04:02.925381 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:02.899335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.3.21.252:44367)
0929 01:04:02.899534 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:02.899542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:02.899610 (+ 68us) server_negotiation.cc:407] Connection header received
0929 01:04:02.899830 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:02.899835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:02.899922 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:02.900026 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:02.901296 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.902694 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.904730 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:02.905587 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:02.905789 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:02.910633 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:02.910641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:02.910646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:02.910696 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:02.919545 (+ 8849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.920563 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.920569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.920572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.921606 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:02.922122 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:02.922128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:02.922132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:02.922740 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:02.922923 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:02.924584 (+ 1661us) server_negotiation.cc:299] Negotiation successful
0929 01:04:02.924898 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:04:02.901671 16776 master_runner.cc:186] Error getting master registration for 127.3.21.253:34941: OK, instance_id { permanent_uuid: "277622c0c7cd4ee4975767f722182356" instance_seqno: 1790643842152865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:04:02.926421 16776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:04:02.931813 16768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: Bootstrap starting.
I20260929 01:04:02.935822 16768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:02.937031 16768 log.cc:824] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:02.940209 16768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: No bootstrap required, opened a new log
I20260929 01:04:02.949702 16768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.950098 16768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:02.950263 16768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277622c0c7cd4ee4975767f722182356, State: Initialized, Role: FOLLOWER
I20260929 01:04:02.951093 16768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.953738 16768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:02.956969 16886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:02.957712 16886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:04:02.959941 16797 tablet.cc:2369] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:02.961601 16851 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:02.961885 16851 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.963647 16851 leader_election.cc:288] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 277622c0c7cd4ee4975767f722182356 (127.3.21.253:34941), a3e92f814f76422f8418bc04777d10cd (127.3.21.252:44367)
I20260929 01:04:02.964783 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5c021558e4746e48dd0efbbd19f6185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277622c0c7cd4ee4975767f722182356" is_pre_election: true
I20260929 01:04:02.964936 16818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5c021558e4746e48dd0efbbd19f6185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e92f814f76422f8418bc04777d10cd" is_pre_election: true
I20260929 01:04:02.965425 16743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5c021558e4746e48dd0efbbd19f6185 in term 0.
I20260929 01:04:02.965706 16818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5c021558e4746e48dd0efbbd19f6185 in term 0.
I20260929 01:04:02.966135 16646 leader_election.cc:302] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277622c0c7cd4ee4975767f722182356, c5c021558e4746e48dd0efbbd19f6185; no voters:
I20260929 01:04:02.966629 16851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:02.966782 16851 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:02.966892 16851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:02.970243 16851 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.971483 16818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5c021558e4746e48dd0efbbd19f6185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e92f814f76422f8418bc04777d10cd"
I20260929 01:04:02.971719 16818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:02.975319 16818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5c021558e4746e48dd0efbbd19f6185 in term 1.
I20260929 01:04:02.975318 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5c021558e4746e48dd0efbbd19f6185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277622c0c7cd4ee4975767f722182356"
I20260929 01:04:02.975634 16743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:02.975855 16643 leader_election.cc:302] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e92f814f76422f8418bc04777d10cd, c5c021558e4746e48dd0efbbd19f6185; no voters:
I20260929 01:04:02.978559 16851 leader_election.cc:288] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [CANDIDATE]: Term 1 election: Requested vote from peers 277622c0c7cd4ee4975767f722182356 (127.3.21.253:34941), a3e92f814f76422f8418bc04777d10cd (127.3.21.252:44367)
I20260929 01:04:02.978794 16851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:02.978928 16851 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [term 1 LEADER]: Becoming Leader. State: Replica: c5c021558e4746e48dd0efbbd19f6185, State: Running, Role: LEADER
I20260929 01:04:02.979535 16851 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } }
I20260929 01:04:02.983338 16743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5c021558e4746e48dd0efbbd19f6185 in term 1.
I20260929 01:04:02.991036 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c021558e4746e48dd0efbbd19f6185. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:02.991894 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:03.000216 16899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:03.000277 16900 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:04:03.003216 16899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:03.024039 16818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [term 1 FOLLOWER]: Refusing update from remote peer c5c021558e4746e48dd0efbbd19f6185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:03.025166 16851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:03.025430 16743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [term 1 FOLLOWER]: Refusing update from remote peer c5c021558e4746e48dd0efbbd19f6185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:03.026436 16851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:03.052281 16870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c021558e4746e48dd0efbbd19f6185. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:03.052618 16870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:03.055531 16901 mvcc.cc:205] Tried to move back new op lower bound from 7334477181012803584 to 7334477180861698048. Current Snapshot: MvccSnapshot[applied={T|T < 7334477181012803584}]
I20260929 01:04:03.063664 16897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:03.064247 16897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:03.065951 16899 catalog_manager.cc:1347] Generated new cluster ID: 97efe4a275e4452b81c99e3a572841d3
I20260929 01:04:03.066061 16899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:03.068058 16886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c021558e4746e48dd0efbbd19f6185. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:03.068441 16886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:03.079449 16908 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:03.080067 16908 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:03.091801 16886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:03.092098 16886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:03.096737 16870 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5c021558e4746e48dd0efbbd19f6185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c021558e4746e48dd0efbbd19f6185" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 41231 } } peers { permanent_uuid: "277622c0c7cd4ee4975767f722182356" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 34941 } } peers { permanent_uuid: "a3e92f814f76422f8418bc04777d10cd" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 44367 } } }
I20260929 01:04:03.097172 16870 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:03.100133 16899 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:03.102062 16899 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:03.124716 16899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5c021558e4746e48dd0efbbd19f6185: Generated new TSK 0
I20260929 01:04:03.126094 16899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:04:03.927202 16883 catalog_manager.cc:1259] Loaded cluster ID: 97efe4a275e4452b81c99e3a572841d3
I20260929 01:04:03.927309 16883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: loading cluster ID for follower catalog manager: success
I20260929 01:04:03.930110 16883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: acquiring CA information for follower catalog manager: success
I20260929 01:04:03.932461 16883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3e92f814f76422f8418bc04777d10cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:04:04.006635 16900 catalog_manager.cc:1259] Loaded cluster ID: 97efe4a275e4452b81c99e3a572841d3
I20260929 01:04:04.006904 16900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: loading cluster ID for follower catalog manager: success
I20260929 01:04:04.010145 16900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: acquiring CA information for follower catalog manager: success
I20260929 01:04:04.012584 16900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 277622c0c7cd4ee4975767f722182356: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:04:04.225494 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16627
I20260929 01:04:04.226989 16689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:04.345074 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16627
I20260929 01:04:04.368808 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16701
I20260929 01:04:04.370234 16763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:04.492242 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16701
I20260929 01:04:04.513029 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16776
I20260929 01:04:04.514285 16838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:04.630362 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16776
2026-09-29T01:04:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4924 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:04:04 dist-test-slave-k0b7 krb5kdc[16920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:04:04 dist-test-slave-k0b7 krb5kdc[16920](info): set up 2 sockets
Sep 29 01:04:04 dist-test-slave-k0b7 krb5kdc[16920](info): commencing operation
krb5kdc: starting...
W20260929 01:04:05.815212 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:04:05 dist-test-slave-k0b7 krb5kdc[16920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:04:06.087692 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37993
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:06.455536 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:06.455866 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:06.455960 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:06.463887 16933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:06.464020 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:06.464071 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:06.464123 16933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:06.464167 16933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:06.474937 16933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:06.476265 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:06.477553 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:06.491695 16938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:06.494374 16939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:06.495954 16941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:06.500609 16933 server_base.cc:1034] running on GCE node
I20260929 01:04:06.501637 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:06.503396 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:06.504696 16933 hybrid_clock.cc:648] HybridClock initialized: now 1790643846504531 us; error 153 us; skew 500 ppm
I20260929 01:04:06.505229 16933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:06 dist-test-slave-k0b7 krb5kdc[16920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643846, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:06.508796 16933 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:06.510617 16933 webserver.cc:460] Webserver started at http://127.3.21.254:32843/ using document root <none> and password file <none>
I20260929 01:04:06.511368 16933 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:06.511476 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:06.511768 16933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:06.514962 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "64a5f57313e34d2ca10377b941735329"
format_stamp: "Formatted at 2026-09-29 01:04:06 on dist-test-slave-k0b7"
server_key: "892d5a501240716419005927d850de91"
server_key_iv: "9ea20868017a6d5e854b0658b1911200"
server_key_version: "encryptionkey@0"
I20260929 01:04:06.515723 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "64a5f57313e34d2ca10377b941735329"
format_stamp: "Formatted at 2026-09-29 01:04:06 on dist-test-slave-k0b7"
server_key: "892d5a501240716419005927d850de91"
server_key_iv: "9ea20868017a6d5e854b0658b1911200"
server_key_version: "encryptionkey@0"
I20260929 01:04:06.521651 16933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 01:04:06.525934 16948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:06.527678 16933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:06.527848 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "64a5f57313e34d2ca10377b941735329"
format_stamp: "Formatted at 2026-09-29 01:04:06 on dist-test-slave-k0b7"
server_key: "892d5a501240716419005927d850de91"
server_key_iv: "9ea20868017a6d5e854b0658b1911200"
server_key_version: "encryptionkey@0"
I20260929 01:04:06.528070 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:06.555969 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:06.561707 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:06.561975 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:06.583847 16933 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:37993
I20260929 01:04:06.583892 16999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:37993 every 8 connection(s)
I20260929 01:04:06.586192 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:06.590895 17000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:06.595826 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 16933
I20260929 01:04:06.596088 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:04:06.596918 3159 external_mini_cluster.cc:1428] Setting key a307707a386a5b4e332a730df27af4bb
I20260929 01:04:06.604166 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329: Bootstrap starting.
I20260929 01:04:06.608321 17000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:06.609683 17000 log.cc:824] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:06.612910 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329: No bootstrap required, opened a new log
Sep 29 01:04:06 dist-test-slave-k0b7 krb5kdc[16920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:06.621299 17000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a5f57313e34d2ca10377b941735329" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37993 } }
I20260929 01:04:06.621733 17000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:06.621873 17000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a5f57313e34d2ca10377b941735329, State: Initialized, Role: FOLLOWER
I20260929 01:04:06.622648 17000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a5f57313e34d2ca10377b941735329" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37993 } }
I20260929 01:04:06.622910 17000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:06.623090 17000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:06.623281 17000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:06.623819 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:06.599283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.3.21.254:37993)
0929 01:04:06.600519 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:06.600532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:06.600584 (+ 52us) server_negotiation.cc:407] Connection header received
0929 01:04:06.602622 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:06.602650 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:06.603437 (+ 787us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:06.603831 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:06.605985 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:06.607446 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:06.609457 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:06.610543 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:06.610762 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:06.615349 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:06.615358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:06.615375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:06.615420 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:06.619062 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:06.619877 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:06.619888 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:06.619896 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:06.620105 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:06.620525 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:06.620531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:06.620544 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:06.621119 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:06.621433 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:06.621926 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 01:04:06.622439 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":718,"threads_started":1}
I20260929 01:04:06.626395 17000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a5f57313e34d2ca10377b941735329" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37993 } }
I20260929 01:04:06.627053 17000 leader_election.cc:302] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a5f57313e34d2ca10377b941735329; no voters:
I20260929 01:04:06.627705 17000 leader_election.cc:288] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:04:06.627853 17005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:06.628628 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:04:06.630331 17000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:06.630857 17005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [term 1 LEADER]: Becoming Leader. State: Replica: 64a5f57313e34d2ca10377b941735329, State: Running, Role: LEADER
I20260929 01:04:06.631549 17005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a5f57313e34d2ca10377b941735329" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37993 } }
I20260929 01:04:06.636137 17006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a5f57313e34d2ca10377b941735329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a5f57313e34d2ca10377b941735329" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37993 } } }
I20260929 01:04:06.636683 17006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:06.636157 17007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a5f57313e34d2ca10377b941735329. Latest consensus state: current_term: 1 leader_uuid: "64a5f57313e34d2ca10377b941735329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a5f57313e34d2ca10377b941735329" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37993 } } }
I20260929 01:04:06.637444 17007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:06.638953 17017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:06.644446 17017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:06.654692 17017 catalog_manager.cc:1347] Generated new cluster ID: 5ec53404e5924a2ab14b0723de53611b
I20260929 01:04:06.654837 17017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:06.671854 17017 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:06.673369 17017 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:06.688776 17017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a5f57313e34d2ca10377b941735329: Generated new TSK 0
I20260929 01:04:06.689989 17017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:04:06.746876 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 16933
I20260929 01:04:06.748425 16995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:06.860227 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 16933
2026-09-29T01:04:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2227 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:04:06 dist-test-slave-k0b7 krb5kdc[17028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:04:06 dist-test-slave-k0b7 krb5kdc[17028](info): set up 2 sockets
Sep 29 01:04:06 dist-test-slave-k0b7 krb5kdc[17028](info): commencing operation
krb5kdc: starting...
W20260929 01:04:08.051508 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:04:08 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:04:08.325095 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36387
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:08.696992 17041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:08.697312 17041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:08.697467 17041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:08.705399 17041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:08.705531 17041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:08.705616 17041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:08.705703 17041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:08.705785 17041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:08.716480 17041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:08.717940 17041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:08.719349 17041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:08.733631 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:08.733795 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:08.733769 17047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:08.734170 17041 server_base.cc:1034] running on GCE node
I20260929 01:04:08.735459 17041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:08.737325 17041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:08.738603 17041 hybrid_clock.cc:648] HybridClock initialized: now 1790643848738542 us; error 35 us; skew 500 ppm
I20260929 01:04:08.739017 17041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:08 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643848, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:08.742468 17041 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:08.744267 17041 webserver.cc:460] Webserver started at http://127.3.21.254:34419/ using document root <none> and password file <none>
I20260929 01:04:08.745035 17041 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:08.745229 17041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:08.745589 17041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:08.749037 17041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "45461be729e04161b94e81c9855d0b4c"
format_stamp: "Formatted at 2026-09-29 01:04:08 on dist-test-slave-k0b7"
server_key: "b17adf962b9ca9eec0c1c22f2b89dbc9"
server_key_iv: "7d95cb131b84ce1036fd21a9a969ddf6"
server_key_version: "encryptionkey@0"
I20260929 01:04:08.749886 17041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "45461be729e04161b94e81c9855d0b4c"
format_stamp: "Formatted at 2026-09-29 01:04:08 on dist-test-slave-k0b7"
server_key: "b17adf962b9ca9eec0c1c22f2b89dbc9"
server_key_iv: "7d95cb131b84ce1036fd21a9a969ddf6"
server_key_version: "encryptionkey@0"
I20260929 01:04:08.756338 17041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:04:08.760759 17056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:08.762905 17041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:08.763180 17041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "45461be729e04161b94e81c9855d0b4c"
format_stamp: "Formatted at 2026-09-29 01:04:08 on dist-test-slave-k0b7"
server_key: "b17adf962b9ca9eec0c1c22f2b89dbc9"
server_key_iv: "7d95cb131b84ce1036fd21a9a969ddf6"
server_key_version: "encryptionkey@0"
I20260929 01:04:08.763513 17041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:08.786867 17041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:08.793183 17041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:08.793496 17041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:08.815667 17041 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36387
I20260929 01:04:08.815697 17107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36387 every 8 connection(s)
I20260929 01:04:08.817116 17041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:08.821638 17108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:08.824134 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17041
I20260929 01:04:08.824378 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:04:08.825188 3159 external_mini_cluster.cc:1428] Setting key 9b50f5bc01b683c4eaebe80501a3f1e3
I20260929 01:04:08.836244 17108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap starting.
I20260929 01:04:08.840504 17108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:08.842455 17108 log.cc:824] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:04:08 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:08.846807 17108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: No bootstrap required, opened a new log
I20260929 01:04:08.855520 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:08.827730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.3.21.254:36387)
0929 01:04:08.828866 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:08.828883 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:08.829175 (+ 292us) server_negotiation.cc:407] Connection header received
0929 01:04:08.831171 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:08.831196 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:08.831639 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:08.832024 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:08.834083 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:08.835871 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:08.837811 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:08.838885 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:08.839129 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:08.845778 (+ 6649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:08.845797 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:08.845817 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:08.845869 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:08.850065 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:08.851413 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:08.851421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:08.851432 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:08.851737 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:08.852220 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:08.852224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:08.852226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:08.852881 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:08.853283 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:08.853793 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 01:04:08.854304 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":440,"threads_started":1}
I20260929 01:04:08.855479 17108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:08.856114 17108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:08.856292 17108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45461be729e04161b94e81c9855d0b4c, State: Initialized, Role: FOLLOWER
I20260929 01:04:08.856958 17108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:08.857290 17108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:08.857509 17108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:08.857730 17108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:08.860582 17108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:08.861119 17108 leader_election.cc:302] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45461be729e04161b94e81c9855d0b4c; no voters:
I20260929 01:04:08.862174 17108 leader_election.cc:288] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:04:08.862352 17113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:08.863765 17113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 LEADER]: Becoming Leader. State: Replica: 45461be729e04161b94e81c9855d0b4c, State: Running, Role: LEADER
I20260929 01:04:08.864364 17113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:08.864725 17108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:08.869023 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45461be729e04161b94e81c9855d0b4c. Latest consensus state: current_term: 1 leader_uuid: "45461be729e04161b94e81c9855d0b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } } }
I20260929 01:04:08.869627 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:08.869058 17114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45461be729e04161b94e81c9855d0b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } } }
I20260929 01:04:08.870204 17114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:08.871400 17122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:08.877557 17122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:08.888839 17122 catalog_manager.cc:1347] Generated new cluster ID: f84e398fd0624ddaadccbb840876b2ee
I20260929 01:04:08.888993 17122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:08.915766 17122 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:08.917557 17122 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:08.927163 17122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Generated new TSK 0
I20260929 01:04:08.928272 17122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:04:09.004285 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36387
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:09.390117 17134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:09.390446 17134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:09.390643 17134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:09.399322 17134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:09.399463 17134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:09.399592 17134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:04:09.410776 17134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:09.412128 17134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:09.413415 17134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:09.428504 17142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:09.428547 17139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:09.428478 17140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:09.430110 17134 server_base.cc:1034] running on GCE node
I20260929 01:04:09.430941 17134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:09.431998 17134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:09.433233 17134 hybrid_clock.cc:648] HybridClock initialized: now 1790643849433178 us; error 51 us; skew 500 ppm
I20260929 01:04:09.433558 17134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:09 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643849, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:09.437191 17134 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:04:09.439301 17134 webserver.cc:460] Webserver started at http://127.3.21.193:42471/ using document root <none> and password file <none>
I20260929 01:04:09.440089 17134 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:09.440202 17134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:09.440481 17134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:09.443699 17134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "784644fbadd24dd186195d9aecd1506b"
format_stamp: "Formatted at 2026-09-29 01:04:09 on dist-test-slave-k0b7"
server_key: "fe3b6e6eaf4b117419a1001c8ecc0ce2"
server_key_iv: "408a5109d83be7ea28446c6ce719f109"
server_key_version: "encryptionkey@0"
I20260929 01:04:09.444468 17134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "784644fbadd24dd186195d9aecd1506b"
format_stamp: "Formatted at 2026-09-29 01:04:09 on dist-test-slave-k0b7"
server_key: "fe3b6e6eaf4b117419a1001c8ecc0ce2"
server_key_iv: "408a5109d83be7ea28446c6ce719f109"
server_key_version: "encryptionkey@0"
I20260929 01:04:09.450628 17134 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:04:09.454615 17149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:09.456430 17134 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:09.456645 17134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "784644fbadd24dd186195d9aecd1506b"
format_stamp: "Formatted at 2026-09-29 01:04:09 on dist-test-slave-k0b7"
server_key: "fe3b6e6eaf4b117419a1001c8ecc0ce2"
server_key_iv: "408a5109d83be7ea28446c6ce719f109"
server_key_version: "encryptionkey@0"
I20260929 01:04:09.456874 17134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:09.482211 17134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:09.487763 17134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:09.488098 17134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:09.489522 17134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:09.491331 17134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:09.491416 17134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:09.491513 17134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:09.491557 17134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:09.529983 17134 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:42599
I20260929 01:04:09.530107 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:42599 every 8 connection(s)
I20260929 01:04:09.531390 17134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:09.541983 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17134
I20260929 01:04:09.542332 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:04:09.543090 3159 external_mini_cluster.cc:1428] Setting key d411444485613b5e338b2a36a4e626c8
Sep 29 01:04:09 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643849, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:09.557672 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:09.534411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:37893 (local address 127.3.21.254:36387)
0929 01:04:09.534969 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:09.534976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:09.535910 (+ 934us) server_negotiation.cc:407] Connection header received
0929 01:04:09.538307 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:09.538313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:09.538413 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:09.538575 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:09.540521 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:09.541545 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:09.543757 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:09.544194 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:09.544347 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:09.548754 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:09.548765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:09.548774 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:09.548832 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:09.553186 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:09.554607 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:09.554614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:09.554616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:09.554780 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:09.555477 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:09.555482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:09.555484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:09.555794 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:09.555939 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:09.557144 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0929 01:04:09.557396 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":296,"threads_started":1}
I20260929 01:04:09.559473 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:09.534842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:37893)
0929 01:04:09.535731 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:09.535772 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:09.537858 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:09.538877 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:09.538887 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:09.539602 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:09.540243 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:09.540263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:09.541751 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:09.541755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:09.543568 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:09.543582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:09.544456 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:09.544464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:09.544751 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:09.545581 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:09.545612 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:09.548371 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:09.553441 (+ 5070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:09.553451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:09.553469 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:09.554264 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:09.554929 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:09.554936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:09.554942 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:09.555270 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:09.555949 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:09.555963 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:09.556547 (+ 584us) client_negotiation.cc:769] Sending connection context
0929 01:04:09.556906 (+ 359us) client_negotiation.cc:240] Negotiation successful
0929 01:04:09.557614 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":368,"threads_started":1}
I20260929 01:04:09.561460 17262 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36387
I20260929 01:04:09.561853 17262 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:09.562738 17262 heartbeater.cc:511] Master 127.3.21.254:36387 requested a full tablet report, sending...
I20260929 01:04:09.565537 17073 ts_manager.cc:194] Registered new tserver with Master: 784644fbadd24dd186195d9aecd1506b (127.3.21.193:42599)
I20260929 01:04:09.567293 17073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:37893
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:04:09.608563 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36387
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:09.980964 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:09.981297 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:09.981495 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:09.989221 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:09.989362 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:09.989570 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:04:09.999727 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:10.001176 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:10.002807 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:10.018623 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:10.018700 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:10.018661 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:10.020592 17268 server_base.cc:1034] running on GCE node
I20260929 01:04:10.021270 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:10.022400 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:10.023739 17268 hybrid_clock.cc:648] HybridClock initialized: now 1790643850023692 us; error 39 us; skew 500 ppm
I20260929 01:04:10.024260 17268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:10 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:10.035315 17268 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:04:10.037243 17268 webserver.cc:460] Webserver started at http://127.3.21.194:36581/ using document root <none> and password file <none>
I20260929 01:04:10.038009 17268 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:10.038128 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:10.038399 17268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:10.041790 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "0f395dbcd8814cd892a14a44dad4fd16"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "bc89092b151124915a4cc3ba8ec17388"
server_key_iv: "72a3ec1b93a1f2364f4344de40e8a454"
server_key_version: "encryptionkey@0"
I20260929 01:04:10.042745 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "0f395dbcd8814cd892a14a44dad4fd16"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "bc89092b151124915a4cc3ba8ec17388"
server_key_iv: "72a3ec1b93a1f2364f4344de40e8a454"
server_key_version: "encryptionkey@0"
I20260929 01:04:10.048678 17268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:04:10.053483 17283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:10.055311 17268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:10.055508 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0f395dbcd8814cd892a14a44dad4fd16"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "bc89092b151124915a4cc3ba8ec17388"
server_key_iv: "72a3ec1b93a1f2364f4344de40e8a454"
server_key_version: "encryptionkey@0"
I20260929 01:04:10.056041 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:10.104511 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:10.111014 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:10.111443 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:10.112707 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:10.114815 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:10.114919 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:10.115059 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:10.115144 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:10.163483 17268 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:32955
I20260929 01:04:10.163640 17395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:32955 every 8 connection(s)
I20260929 01:04:10.165935 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:10.173394 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17268
I20260929 01:04:10.173601 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:04:10.174203 3159 external_mini_cluster.cc:1428] Setting key 96a323013f3b0ebb7066e990a4eb59a2
Sep 29 01:04:10 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:04:10.205276 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.177743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:58191 (local address 127.3.21.254:36387)
0929 01:04:10.181531 (+ 3788us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:10.181535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:10.181552 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:04:10.181608 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:10.181612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:10.181687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:10.181782 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:10.184584 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.186109 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.188239 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.189258 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.189494 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.194978 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:10.194990 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:10.194999 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:10.195065 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:10.199951 (+ 4886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:10.202006 (+ 2055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:10.202018 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:10.202025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:10.202251 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:10.203066 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:10.203075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:10.203081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:10.203506 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:10.203656 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:10.204736 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 01:04:10.204962 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3650,"thread_start_us":3526,"threads_started":1}
I20260929 01:04:10.206622 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.176046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:58191)
0929 01:04:10.177088 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:10.177141 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:10.180561 (+ 3420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:10.182154 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:10.182172 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:10.183157 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:10.184099 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.184134 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.186341 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.186346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.188011 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.188029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.189723 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.189732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.190068 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.191282 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:10.191320 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:10.194536 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:10.200321 (+ 5785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:10.200334 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:10.200359 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:10.201742 (+ 1383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:10.202461 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:10.202465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:10.202468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:10.202887 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:10.203659 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:10.203671 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:10.204225 (+ 554us) client_negotiation.cc:769] Sending connection context
0929 01:04:10.204577 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 01:04:10.205062 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":470,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:04:10.208590 17396 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36387
I20260929 01:04:10.209116 17396 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:10.210000 17396 heartbeater.cc:511] Master 127.3.21.254:36387 requested a full tablet report, sending...
I20260929 01:04:10.211926 17073 ts_manager.cc:194] Registered new tserver with Master: 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194:32955)
I20260929 01:04:10.212853 17073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:58191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:04:10.249291 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36387
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:10.571043 17262 heartbeater.cc:503] Master 127.3.21.254:36387 was elected leader, sending a full tablet report...
W20260929 01:04:10.642654 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:10.642983 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:10.643131 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:10.651420 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:10.651575 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:10.651798 17402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:04:10.663158 17402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:10.664695 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:10.666278 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:10.682405 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:10.682657 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:10.682766 17408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:10.683714 17402 server_base.cc:1034] running on GCE node
I20260929 01:04:10.684501 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:10.685566 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:10.686792 17402 hybrid_clock.cc:648] HybridClock initialized: now 1790643850686750 us; error 43 us; skew 500 ppm
I20260929 01:04:10.687124 17402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:10 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:10.691025 17402 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:04:10.692919 17402 webserver.cc:460] Webserver started at http://127.3.21.195:33865/ using document root <none> and password file <none>
I20260929 01:04:10.693792 17402 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:10.693918 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:10.694285 17402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:10.698024 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "56a2a419e83c4936a27a0bf2057fc078"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "2f2fb53e218c957ca3ef8de6f200b5f2"
server_key_iv: "7b3fa6a57749c90c2080228408ad2d45"
server_key_version: "encryptionkey@0"
I20260929 01:04:10.699120 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "56a2a419e83c4936a27a0bf2057fc078"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "2f2fb53e218c957ca3ef8de6f200b5f2"
server_key_iv: "7b3fa6a57749c90c2080228408ad2d45"
server_key_version: "encryptionkey@0"
I20260929 01:04:10.706967 17402 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 01:04:10.711220 17417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:10.713053 17402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:04:10.713375 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "56a2a419e83c4936a27a0bf2057fc078"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "2f2fb53e218c957ca3ef8de6f200b5f2"
server_key_iv: "7b3fa6a57749c90c2080228408ad2d45"
server_key_version: "encryptionkey@0"
I20260929 01:04:10.713632 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:10.749588 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:10.755380 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:10.755733 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:10.756971 17402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:10.758577 17402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:10.758690 17402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:10.758846 17402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:10.758929 17402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:10.804217 17402 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37187
I20260929 01:04:10.804240 17529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37187 every 8 connection(s)
I20260929 01:04:10.805612 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:10.812162 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17402
I20260929 01:04:10.812386 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:04:10.812880 3159 external_mini_cluster.cc:1428] Setting key 05059f140ba6bf5689c5a7ccd82a9fd8
Sep 29 01:04:10 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:10.831964 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.808930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:34671 (local address 127.3.21.254:36387)
0929 01:04:10.809486 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:10.809491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:10.810653 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 01:04:10.812697 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:10.812703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:10.812828 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:10.812965 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:10.814892 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.816102 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.818056 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.818617 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.818760 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.823400 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:10.823406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:10.823409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:10.823448 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:10.826936 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:10.828107 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:10.828116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:10.828120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:10.828480 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:10.829581 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:10.829587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:10.829590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:10.829975 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:10.830286 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:10.831225 (+ 939us) server_negotiation.cc:299] Negotiation successful
0929 01:04:10.831548 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
I20260929 01:04:10.833467 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.809597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:34671)
0929 01:04:10.810370 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:10.810414 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:10.812299 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:10.813226 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:10.813236 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:10.813988 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:10.814696 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.814717 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.816277 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.816281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.817888 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.817902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.818862 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.818870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.819140 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.820048 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:10.820074 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:10.823128 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:10.827156 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:10.827165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:10.827181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:10.827743 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:10.828764 (+ 1021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:10.828769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:10.828772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:10.829289 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:10.830203 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:10.830214 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:10.830768 (+ 554us) client_negotiation.cc:769] Sending connection context
0929 01:04:10.831108 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 01:04:10.831746 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":339,"threads_started":1}
I20260929 01:04:10.835836 17530 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36387
I20260929 01:04:10.836400 17530 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:10.837548 17530 heartbeater.cc:511] Master 127.3.21.254:36387 requested a full tablet report, sending...
I20260929 01:04:10.839910 17073 ts_manager.cc:194] Registered new tserver with Master: 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187)
I20260929 01:04:10.840862 17073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:34671
I20260929 01:04:10.844233 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:10.864289 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.849165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.3.21.254:36387)
0929 01:04:10.849439 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:10.849448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:10.849774 (+ 326us) server_negotiation.cc:407] Connection header received
0929 01:04:10.849995 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:10.850001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:10.850126 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:10.850247 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:10.851883 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.852626 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.854183 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.854720 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.854874 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.857478 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:10.857483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:10.857486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:10.857531 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:10.860788 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:10.862102 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:10.862109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:10.862113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:10.862277 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:10.862750 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:10.862758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:10.862763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:10.863118 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:10.863416 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:10.863704 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 01:04:10.863960 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 01:04:10.869984 17073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:04:10.877760 17073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:10.905179 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.892144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:32955 (local address 127.0.0.1:40374)
0929 01:04:10.893428 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:10.893465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:10.893649 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:10.895955 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:10.895964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:10.895998 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:10.896548 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.896568 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.899135 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.899141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.902407 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.902420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.903997 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.904004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.904267 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.904343 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 01:04:10.904478 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:04:10.904688 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":3,"thread_start_us":865,"threads_started":1}
I20260929 01:04:10.905738 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.892343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.3.21.194:32955)
0929 01:04:10.894391 (+ 2048us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:10.894404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:10.894440 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:04:10.894590 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:10.894616 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:10.895673 (+ 1057us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:10.896776 (+ 1103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:10.897033 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.898916 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.902608 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.903740 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.904905 (+ 1165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.905116 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:10.905199 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 01:04:10.905401 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1826,"thread_start_us":1686,"threads_started":1}
I20260929 01:04:10.905247 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.892758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:48486)
0929 01:04:10.895699 (+ 2941us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:10.895722 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:10.895870 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:10.896931 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:10.896936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:10.896972 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:10.897566 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.897582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.899514 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.899519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.901920 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.901936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.904321 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.904328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.904528 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.904577 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 01:04:10.904657 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:04:10.904807 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2800,"mutex_wait_us":46,"thread_start_us":2599,"threads_started":1}
I20260929 01:04:10.906566 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.893045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.3.21.193:42599)
0929 01:04:10.895358 (+ 2313us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:10.895370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:10.896021 (+ 651us) server_negotiation.cc:407] Connection header received
0929 01:04:10.896133 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:10.896141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:10.896418 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:10.896640 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:10.897852 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.899333 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.902097 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.903738 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.903883 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.904155 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:10.905579 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0929 01:04:10.905751 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2108,"thread_start_us":1926,"threads_started":1}
I20260929 01:04:10.907521 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.895739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37187 (local address 127.0.0.1:48974)
0929 01:04:10.897660 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:10.897702 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:10.897917 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:10.898594 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:10.898600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:10.899029 (+ 429us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:10.900224 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.900242 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.902211 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.902216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.904441 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:10.904455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.906696 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.906703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:10.906885 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.906918 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:04:10.907010 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 01:04:10.907157 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809,"mutex_wait_us":35,"thread_start_us":1588,"threads_started":1}
I20260929 01:04:10.908216 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:10.896337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.3.21.195:37187)
0929 01:04:10.897052 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:10.897064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:10.897880 (+ 816us) server_negotiation.cc:407] Connection header received
0929 01:04:10.898101 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:10.898111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:10.898352 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:10.898614 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:10.900443 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.902059 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.904640 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:10.906133 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:10.906245 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:10.906460 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:10.907692 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0929 01:04:10.907915 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":353,"threads_started":1}
I20260929 01:04:10.910116 17331 tablet_service.cc:1467] Processing CreateTablet for tablet 81220da473c248c48a40d0b508850d46 (DEFAULT_TABLE table=test-table [id=659122726b924e70ac6b3aa795b9cf04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:10.910116 17197 tablet_service.cc:1467] Processing CreateTablet for tablet 81220da473c248c48a40d0b508850d46 (DEFAULT_TABLE table=test-table [id=659122726b924e70ac6b3aa795b9cf04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:10.911701 17197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81220da473c248c48a40d0b508850d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:10.911702 17331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81220da473c248c48a40d0b508850d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:10.911602 17465 tablet_service.cc:1467] Processing CreateTablet for tablet 81220da473c248c48a40d0b508850d46 (DEFAULT_TABLE table=test-table [id=659122726b924e70ac6b3aa795b9cf04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:10.912911 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81220da473c248c48a40d0b508850d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:10.925864 17549 tablet_bootstrap.cc:492] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Bootstrap starting.
I20260929 01:04:10.928393 17550 tablet_bootstrap.cc:492] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Bootstrap starting.
I20260929 01:04:10.929349 17551 tablet_bootstrap.cc:492] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Bootstrap starting.
I20260929 01:04:10.930521 17549 tablet_bootstrap.cc:654] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:10.932331 17549 log.cc:824] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:10.933513 17550 tablet_bootstrap.cc:654] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:10.933772 17551 tablet_bootstrap.cc:654] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:10.935271 17550 log.cc:824] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:10.935427 17551 log.cc:824] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:10.936792 17549 tablet_bootstrap.cc:492] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: No bootstrap required, opened a new log
I20260929 01:04:10.937130 17549 ts_tablet_manager.cc:1397] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260929 01:04:10.939152 17550 tablet_bootstrap.cc:492] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: No bootstrap required, opened a new log
I20260929 01:04:10.939458 17550 ts_tablet_manager.cc:1397] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 01:04:10.940512 17551 tablet_bootstrap.cc:492] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: No bootstrap required, opened a new log
I20260929 01:04:10.940794 17551 ts_tablet_manager.cc:1397] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 01:04:10.947981 17550 raft_consensus.cc:348] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:10.948448 17550 raft_consensus.cc:374] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:10.948565 17550 raft_consensus.cc:729] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f395dbcd8814cd892a14a44dad4fd16, State: Initialized, Role: FOLLOWER
I20260929 01:04:10.948611 17549 raft_consensus.cc:348] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:10.949136 17549 raft_consensus.cc:374] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:10.949252 17549 raft_consensus.cc:729] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a2a419e83c4936a27a0bf2057fc078, State: Initialized, Role: FOLLOWER
I20260929 01:04:10.949282 17550 consensus_queue.cc:260] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:10.949877 17549 consensus_queue.cc:260] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:10.950698 17396 heartbeater.cc:503] Master 127.3.21.254:36387 was elected leader, sending a full tablet report...
I20260929 01:04:10.951498 17550 ts_tablet_manager.cc:1428] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 01:04:10.952301 17551 raft_consensus.cc:348] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:10.952616 17530 heartbeater.cc:503] Master 127.3.21.254:36387 was elected leader, sending a full tablet report...
I20260929 01:04:10.952873 17551 raft_consensus.cc:374] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:10.953181 17551 raft_consensus.cc:729] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784644fbadd24dd186195d9aecd1506b, State: Initialized, Role: FOLLOWER
I20260929 01:04:10.953289 17549 ts_tablet_manager.cc:1428] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260929 01:04:10.953965 17551 consensus_queue.cc:260] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:10.956094 17551 ts_tablet_manager.cc:1428] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
W20260929 01:04:11.034415 17263 tablet.cc:2369] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:11.057700 17531 tablet.cc:2369] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:11.156253 17555 raft_consensus.cc:482] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:11.156548 17555 raft_consensus.cc:504] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:11.158366 17555 leader_election.cc:288] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 784644fbadd24dd186195d9aecd1506b (127.3.21.193:42599), 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187)
I20260929 01:04:11.168689 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.159050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:47419 (local address 127.3.21.193:42599)
0929 01:04:11.159356 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:11.159366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:11.160143 (+ 777us) server_negotiation.cc:407] Connection header received
0929 01:04:11.160509 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:11.160516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:11.160609 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:11.160671 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:11.161816 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.162944 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.165885 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.167396 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.167648 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.167740 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:11.168138 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 01:04:11.168292 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:04:11.169000 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.158942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.3.21.194:47419)
0929 01:04:11.160004 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.160027 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:11.160270 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:11.160944 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:11.160950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:11.160991 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:11.161633 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:11.161647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.163087 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.163095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:11.165666 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:11.165684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.167707 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.167717 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:11.167931 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.167967 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:04:11.168109 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:04:11.168290 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":28,"thread_start_us":534,"threads_started":1}
I20260929 01:04:11.169157 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.160278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:53111 (local address 127.3.21.195:37187)
0929 01:04:11.160501 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:11.160508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:11.160896 (+ 388us) server_negotiation.cc:407] Connection header received
0929 01:04:11.161043 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:11.161049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:11.161140 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:11.161249 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:11.162460 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.163751 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.166541 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.167466 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.167727 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.167814 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:11.168565 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 01:04:11.168807 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:04:11.170135 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.160040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37187 (local address 127.3.21.194:53111)
0929 01:04:11.160766 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.160781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:11.160897 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:11.161309 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:11.161321 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:11.161354 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:11.162259 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:11.162274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.163917 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.163923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:11.166321 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:11.166334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.167955 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.167969 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:11.168282 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.168322 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:04:11.169502 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0929 01:04:11.169671 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1,"thread_start_us":409,"threads_started":1}
I20260929 01:04:11.170364 17217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81220da473c248c48a40d0b508850d46" candidate_uuid: "0f395dbcd8814cd892a14a44dad4fd16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784644fbadd24dd186195d9aecd1506b" is_pre_election: true
I20260929 01:04:11.171059 17217 raft_consensus.cc:2454] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f395dbcd8814cd892a14a44dad4fd16 in term 0.
I20260929 01:04:11.171159 17485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81220da473c248c48a40d0b508850d46" candidate_uuid: "0f395dbcd8814cd892a14a44dad4fd16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a2a419e83c4936a27a0bf2057fc078" is_pre_election: true
I20260929 01:04:11.171666 17485 raft_consensus.cc:2454] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f395dbcd8814cd892a14a44dad4fd16 in term 0.
I20260929 01:04:11.171905 17285 leader_election.cc:302] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f395dbcd8814cd892a14a44dad4fd16, 784644fbadd24dd186195d9aecd1506b; no voters:
I20260929 01:04:11.172427 17555 raft_consensus.cc:2781] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:11.172612 17555 raft_consensus.cc:482] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:11.172724 17555 raft_consensus.cc:3037] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:04:11.175575 17397 tablet.cc:2369] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:11.176789 17555 raft_consensus.cc:504] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:11.177811 17555 leader_election.cc:288] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [CANDIDATE]: Term 1 election: Requested vote from peers 784644fbadd24dd186195d9aecd1506b (127.3.21.193:42599), 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187)
I20260929 01:04:11.178431 17217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81220da473c248c48a40d0b508850d46" candidate_uuid: "0f395dbcd8814cd892a14a44dad4fd16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784644fbadd24dd186195d9aecd1506b"
I20260929 01:04:11.178439 17485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81220da473c248c48a40d0b508850d46" candidate_uuid: "0f395dbcd8814cd892a14a44dad4fd16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a2a419e83c4936a27a0bf2057fc078"
I20260929 01:04:11.178705 17217 raft_consensus.cc:3037] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:11.178742 17485 raft_consensus.cc:3037] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:11.181187 17485 raft_consensus.cc:2454] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f395dbcd8814cd892a14a44dad4fd16 in term 1.
I20260929 01:04:11.181751 17285 leader_election.cc:302] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f395dbcd8814cd892a14a44dad4fd16, 56a2a419e83c4936a27a0bf2057fc078; no voters:
I20260929 01:04:11.182027 17217 raft_consensus.cc:2454] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f395dbcd8814cd892a14a44dad4fd16 in term 1.
I20260929 01:04:11.182114 17555 raft_consensus.cc:2781] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:11.182840 17555 raft_consensus.cc:686] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 LEADER]: Becoming Leader. State: Replica: 0f395dbcd8814cd892a14a44dad4fd16, State: Running, Role: LEADER
I20260929 01:04:11.183370 17555 consensus_queue.cc:237] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:11.188570 17071 catalog_manager.cc:5495] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194). New cstate: current_term: 1 leader_uuid: "0f395dbcd8814cd892a14a44dad4fd16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } health_report { overall_health: UNKNOWN } } }
I20260929 01:04:11.279718 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.273055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.3.21.194:32955)
0929 01:04:11.273257 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:11.273262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:11.273281 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:04:11.273508 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:11.273514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:11.273611 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:11.273781 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:11.274418 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.275392 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.276981 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.277455 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.277657 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.278248 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:11.278936 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:11.279155 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:11.279403 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 01:04:11.279484 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:04:11.289222 17485 raft_consensus.cc:1261] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Refusing update from remote peer 0f395dbcd8814cd892a14a44dad4fd16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:11.289358 17217 raft_consensus.cc:1261] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Refusing update from remote peer 0f395dbcd8814cd892a14a44dad4fd16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:11.290407 17560 consensus_queue.cc:1035] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:11.290915 17555 consensus_queue.cc:1035] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:11.325539 17070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:11.325847 17070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:11.330938 17070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Sending DeleteTablet for 3 replicas of tablet 81220da473c248c48a40d0b508850d46
I20260929 01:04:11.332641 17331 tablet_service.cc:1514] Processing DeleteTablet for tablet 81220da473c248c48a40d0b508850d46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:11 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:40374
I20260929 01:04:11.332654 17197 tablet_service.cc:1514] Processing DeleteTablet for tablet 81220da473c248c48a40d0b508850d46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:11 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48486
I20260929 01:04:11.333040 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17134
I20260929 01:04:11.335705 17465 tablet_service.cc:1514] Processing DeleteTablet for tablet 81220da473c248c48a40d0b508850d46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:11 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:48974
I20260929 01:04:11.360083 17576 tablet_replica.cc:330] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: stopping tablet replica
I20260929 01:04:11.365337 17576 raft_consensus.cc:2229] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:11.366708 17576 raft_consensus.cc:2258] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:11.366986 17574 tablet_replica.cc:330] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: stopping tablet replica
I20260929 01:04:11.367578 17574 raft_consensus.cc:2229] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:04:11.368420 17574 raft_consensus.cc:2258] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:11.370308 17573 tablet_replica.cc:330] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: stopping tablet replica
I20260929 01:04:11.370886 17573 raft_consensus.cc:2229] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:11.371546 17576 ts_tablet_manager.cc:1905] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:11.371798 17574 ts_tablet_manager.cc:1905] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:11.372149 17573 raft_consensus.cc:2258] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 01:04:11 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:04:11.380178 17574 ts_tablet_manager.cc:1918] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:11.380501 17574 log.cc:1196] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/81220da473c248c48a40d0b508850d46
I20260929 01:04:11.381182 17574 ts_tablet_manager.cc:1939] T 81220da473c248c48a40d0b508850d46 P 0f395dbcd8814cd892a14a44dad4fd16: Deleting consensus metadata
I20260929 01:04:11.382975 17576 ts_tablet_manager.cc:1918] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:11.383189 17576 log.cc:1196] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/81220da473c248c48a40d0b508850d46
I20260929 01:04:11.383373 17057 catalog_manager.cc:4841] TS 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194:32955): tablet 81220da473c248c48a40d0b508850d46 (table test-table [id=659122726b924e70ac6b3aa795b9cf04]) successfully deleted
I20260929 01:04:11.383800 17576 ts_tablet_manager.cc:1939] T 81220da473c248c48a40d0b508850d46 P 56a2a419e83c4936a27a0bf2057fc078: Deleting consensus metadata
I20260929 01:04:11.381461 17573 ts_tablet_manager.cc:1905] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:11.385984 17058 catalog_manager.cc:4841] TS 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187): tablet 81220da473c248c48a40d0b508850d46 (table test-table [id=659122726b924e70ac6b3aa795b9cf04]) successfully deleted
I20260929 01:04:11.389652 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.336752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.3.21.193:42599)
0929 01:04:11.337021 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:11.337032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:11.359421 (+ 22389us) server_negotiation.cc:407] Connection header received
0929 01:04:11.361795 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:11.361806 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:11.362000 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:11.362142 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:11.369660 (+ 7518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.370913 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.373212 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.373905 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.374015 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.381284 (+ 7269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:11.381302 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:11.381320 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:11.381376 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:11.385891 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:11.386953 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:11.386962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:11.386965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:11.387140 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:11.387748 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:11.387752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:11.387754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:11.388210 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:11.388623 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:11.388933 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 01:04:11.389175 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 01:04:11.390580 17257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:11.490705 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17134
W20260929 01:04:11.516914 17058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.21.193:42599 (error 108)
I20260929 01:04:11.517244 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17268
I20260929 01:04:11.520279 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.519511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:48494)
0929 01:04:11.520002 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.520113 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":258,"threads_started":1}
W20260929 01:04:11.520594 17058 catalog_manager.cc:4584] TS 784644fbadd24dd186195d9aecd1506b (127.3.21.193:42599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81220da473c248c48a40d0b508850d46: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Sep 29 01:04:11 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:04:11.538791 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.518173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.3.21.194:32955)
0929 01:04:11.518716 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:11.518722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:11.519021 (+ 299us) server_negotiation.cc:407] Connection header received
0929 01:04:11.519093 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:11.519097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:11.519235 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:11.519361 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:11.521176 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.521990 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.525299 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.525854 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.526031 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.532050 (+ 6019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:11.532070 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:11.532091 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:11.532143 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:11.535524 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:11.536418 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:11.536425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:11.536428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:11.536602 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:11.537026 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:11.537033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:11.537035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:11.537401 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:11.537787 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:11.538100 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 01:04:11.538350 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393}
I20260929 01:04:11.539641 17391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:11.574556 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.573966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:48510)
0929 01:04:11.574238 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.574356 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 01:04:11.576000 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.575517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:48526)
0929 01:04:11.575745 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.575824 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 01:04:11.636649 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.636121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:41256)
0929 01:04:11.636363 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.636458 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 01:04:11.637724 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.637337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:41270)
0929 01:04:11.637490 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.637580 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:04:11.662658 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17268
I20260929 01:04:11.682993 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17402
Sep 29 01:04:11 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:04:11.701730 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.683933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.3.21.195:37187)
0929 01:04:11.684557 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:11.684562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:11.684579 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:04:11.684636 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:11.684639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:11.684703 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:11.684830 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:11.686389 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.687417 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.688847 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:11.689402 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:11.689556 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:11.694750 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:11.694771 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:11.694788 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:11.694833 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:11.698035 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:11.699011 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:11.699020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:11.699023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:11.699269 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:11.699663 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:11.699666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:11.699669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:11.700165 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:11.700531 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:11.701065 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 01:04:11.701333 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":399,"threads_started":1}
I20260929 01:04:11.702407 17525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:11.718750 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.718112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:41272)
0929 01:04:11.718355 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.718556 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 01:04:11.719978 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.719489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:41286)
0929 01:04:11.719714 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.719804 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 01:04:11.798664 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17402
I20260929 01:04:11.816982 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17041
I20260929 01:04:11.818346 17103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:11.939272 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:11.937529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:41290)
0929 01:04:11.938694 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:11.938883 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611}
I20260929 01:04:11.939456 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17041
I20260929 01:04:11.961412 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36387
--webserver_interface=127.3.21.254
--webserver_port=34419
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:12.333164 17584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:12.333598 17584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:12.333797 17584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:12.341970 17584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:12.342193 17584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:12.342337 17584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:12.342451 17584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:12.342582 17584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:12.354652 17584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:12.356215 17584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:12.357707 17584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:12.382956 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:12.382947 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:12.382884 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:12.383723 17584 server_base.cc:1034] running on GCE node
I20260929 01:04:12.384742 17584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:12.386580 17584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:12.387878 17584 hybrid_clock.cc:648] HybridClock initialized: now 1790643852387771 us; error 91 us; skew 500 ppm
I20260929 01:04:12.388640 17584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:12 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:12.392635 17584 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:12.394626 17584 webserver.cc:460] Webserver started at http://127.3.21.254:34419/ using document root <none> and password file <none>
I20260929 01:04:12.395494 17584 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:12.395680 17584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:12.402824 17584 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:04:12.406725 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:12.408910 17584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:12.409233 17584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "45461be729e04161b94e81c9855d0b4c"
format_stamp: "Formatted at 2026-09-29 01:04:08 on dist-test-slave-k0b7"
server_key: "b17adf962b9ca9eec0c1c22f2b89dbc9"
server_key_iv: "7d95cb131b84ce1036fd21a9a969ddf6"
server_key_version: "encryptionkey@0"
I20260929 01:04:12.410683 17584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:12.426411 17584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:12.432314 17584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:12.432677 17584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:12.456086 17584 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36387
I20260929 01:04:12.456188 17651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36387 every 8 connection(s)
I20260929 01:04:12.457892 17584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:12.458865 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17584
I20260929 01:04:12.460023 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:42599
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:36387
--webserver_port=42471
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:12.467362 17652 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:04:12.471449 17652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap starting.
I20260929 01:04:12.494671 17652 log.cc:824] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:12.508481 17652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:12.509130 17652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap complete.
I20260929 01:04:12.520717 17652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:12.522007 17652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45461be729e04161b94e81c9855d0b4c, State: Initialized, Role: FOLLOWER
I20260929 01:04:12.522938 17652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:12.523231 17652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:12.523372 17652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:12.523561 17652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:04:12.527846 17652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:12.528461 17652 leader_election.cc:302] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45461be729e04161b94e81c9855d0b4c; no voters:
I20260929 01:04:12.529177 17652 leader_election.cc:288] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:04:12.529328 17656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:04:12.530730 17656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 LEADER]: Becoming Leader. State: Replica: 45461be729e04161b94e81c9855d0b4c, State: Running, Role: LEADER
I20260929 01:04:12.531495 17656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:12.531875 17652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:12.534118 17658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45461be729e04161b94e81c9855d0b4c. Latest consensus state: current_term: 2 leader_uuid: "45461be729e04161b94e81c9855d0b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } } }
I20260929 01:04:12.535676 17658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:12.534874 17657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45461be729e04161b94e81c9855d0b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } } }
I20260929 01:04:12.536087 17657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:12.543648 17663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:12.554123 17663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:12.555814 17663 catalog_manager.cc:1259] Loaded cluster ID: f84e398fd0624ddaadccbb840876b2ee
I20260929 01:04:12.556152 17663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:12.560027 17663 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 01:04:12.578997 17663 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 01:04:12.594951 17663 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:12.596946 17663 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Loaded TSK: 0
I20260929 01:04:12.599248 17663 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:04:12.862910 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:12.863318 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:12.863497 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:12.871364 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:12.871543 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:12.871739 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:04:12.882453 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=42471
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:12.884124 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:12.885684 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:12.904479 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:12.904740 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:12.906419 17654 server_base.cc:1034] running on GCE node
W20260929 01:04:12.906832 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:12.907757 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:12.909022 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:12.910310 17654 hybrid_clock.cc:648] HybridClock initialized: now 1790643852910257 us; error 49 us; skew 500 ppm
I20260929 01:04:12.910745 17654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:12 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:12.914837 17654 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:04:12.917155 17654 webserver.cc:460] Webserver started at http://127.3.21.193:42471/ using document root <none> and password file <none>
I20260929 01:04:12.918030 17654 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:12.918243 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:12.925436 17654 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:04:12.929102 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:12.930862 17654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:04:12.931067 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "784644fbadd24dd186195d9aecd1506b"
format_stamp: "Formatted at 2026-09-29 01:04:09 on dist-test-slave-k0b7"
server_key: "fe3b6e6eaf4b117419a1001c8ecc0ce2"
server_key_iv: "408a5109d83be7ea28446c6ce719f109"
server_key_version: "encryptionkey@0"
I20260929 01:04:12.932159 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:12.952421 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:12.957736 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:12.958211 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:12.959616 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:12.961879 17696 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 01:04:12.967826 17696 ts_tablet_manager.cc:1852] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 01:04:12.968956 17696 ts_tablet_manager.cc:1905] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:12.976979 17696 ts_tablet_manager.cc:1918] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:12.977248 17696 log.cc:1196] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/81220da473c248c48a40d0b508850d46
I20260929 01:04:12.977798 17696 ts_tablet_manager.cc:1939] T 81220da473c248c48a40d0b508850d46 P 784644fbadd24dd186195d9aecd1506b: Deleting consensus metadata
I20260929 01:04:12.978276 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 01:04:12.978376 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260929 01:04:12.978574 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:12.978683 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:13.020571 17654 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:42599
I20260929 01:04:13.020658 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:42599 every 8 connection(s)
I20260929 01:04:13.021992 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:13.028330 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17654
I20260929 01:04:13.029106 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:32955
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:36387
--webserver_port=36581
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:13 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:13.052333 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:13.025343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:55831 (local address 127.3.21.254:36387)
0929 01:04:13.026037 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:13.026059 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:13.026844 (+ 785us) server_negotiation.cc:407] Connection header received
0929 01:04:13.030248 (+ 3404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:13.030262 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:13.030806 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:13.031173 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:13.033649 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.035079 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.037444 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.038140 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.038369 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:13.042303 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:13.042313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:13.042330 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:13.042383 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:13.047198 (+ 4815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:13.048264 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:13.048272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:13.048278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:13.048473 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:13.048949 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:13.048956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:13.048959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:13.049475 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:13.049684 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:13.050546 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 01:04:13.050965 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260929 01:04:13.052333 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:13.025622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:55831)
0929 01:04:13.026633 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:13.026701 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:13.029369 (+ 2668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:13.031415 (+ 2046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:13.031429 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:13.032316 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:13.033249 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:13.033287 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.035400 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.035409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:13.037197 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:13.037244 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.038415 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.038422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:13.038741 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:13.039494 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:13.039517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:13.041954 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:13.047422 (+ 5468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:13.047437 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:13.047452 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:13.048091 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:13.048630 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:13.048634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:13.048636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:13.048823 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:13.049638 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:13.049647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:13.050001 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 01:04:13.050287 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 01:04:13.050927 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":374,"threads_started":1}
I20260929 01:04:13.056309 17803 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36387
I20260929 01:04:13.056789 17803 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:13.057658 17803 heartbeater.cc:511] Master 127.3.21.254:36387 requested a full tablet report, sending...
I20260929 01:04:13.060086 17617 ts_manager.cc:194] Registered new tserver with Master: 784644fbadd24dd186195d9aecd1506b (127.3.21.193:42599)
I20260929 01:04:13.061616 17617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:55831
W20260929 01:04:13.401389 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:13.401715 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:13.401872 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:13.409687 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:13.409832 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:13.410046 17807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:04:13.420781 17807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:32955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=36581
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:13.422724 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:13.424206 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:13.440663 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:13.440752 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:13.440719 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:13.441779 17807 server_base.cc:1034] running on GCE node
I20260929 01:04:13.442553 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:13.443754 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:13.445113 17807 hybrid_clock.cc:648] HybridClock initialized: now 1790643853445038 us; error 74 us; skew 500 ppm
I20260929 01:04:13.445557 17807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:13 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:13.449755 17807 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:04:13.451684 17807 webserver.cc:460] Webserver started at http://127.3.21.194:36581/ using document root <none> and password file <none>
I20260929 01:04:13.452533 17807 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:13.452728 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:13.459188 17807 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:04:13.462759 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:13.464457 17807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:13.464700 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0f395dbcd8814cd892a14a44dad4fd16"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "bc89092b151124915a4cc3ba8ec17388"
server_key_iv: "72a3ec1b93a1f2364f4344de40e8a454"
server_key_version: "encryptionkey@0"
I20260929 01:04:13.465938 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:13.489795 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:13.494652 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:13.494947 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:13.496047 17807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:13.497713 17807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:13.497859 17807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:13.498018 17807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:13.498109 17807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:13.535264 17807 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:32955
I20260929 01:04:13.535357 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:32955 every 8 connection(s)
I20260929 01:04:13.536842 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:13.538527 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17807
I20260929 01:04:13.539335 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:37187
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:36387
--webserver_port=33865
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:13 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:13.561614 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:13.539947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:57445 (local address 127.3.21.254:36387)
0929 01:04:13.540316 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:13.540321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:13.541651 (+ 1330us) server_negotiation.cc:407] Connection header received
0929 01:04:13.543747 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:13.543753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:13.543863 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:13.543993 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:13.545904 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.546917 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.548587 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.549117 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.549262 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:13.553456 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:13.553462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:13.553465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:13.553504 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:13.556971 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:13.558069 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:13.558075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:13.558079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:13.558252 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:13.559004 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:13.559013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:13.559019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:13.559508 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:13.559705 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:13.560798 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0929 01:04:13.561122 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260929 01:04:13.562794 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:13.540611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:57445)
0929 01:04:13.541415 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:13.541459 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:13.543417 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:13.544262 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:13.544270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:13.544999 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:13.545676 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:13.545707 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.547101 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.547113 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:13.548415 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:13.548428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:13.549389 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:13.549397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:13.549640 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:13.550408 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:13.550437 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:13.553173 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:13.557281 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:13.557291 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:13.557320 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:13.557910 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:13.558430 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:13.558438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:13.558445 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:13.558816 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:13.559710 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:13.559717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:13.560240 (+ 523us) client_negotiation.cc:769] Sending connection context
0929 01:04:13.560624 (+ 384us) client_negotiation.cc:240] Negotiation successful
0929 01:04:13.561320 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":318,"threads_started":1}
I20260929 01:04:13.564452 17935 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36387
I20260929 01:04:13.564823 17935 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:13.565613 17935 heartbeater.cc:511] Master 127.3.21.254:36387 requested a full tablet report, sending...
I20260929 01:04:13.567298 17617 ts_manager.cc:194] Registered new tserver with Master: 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194:32955)
I20260929 01:04:13.568171 17617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:57445
W20260929 01:04:13.898906 17938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:13.899260 17938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:13.899432 17938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:13.907729 17938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:13.907924 17938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:13.908111 17938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:04:13.919080 17938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:37187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=33865
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:13.920645 17938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:13.922089 17938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:13.937615 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:13.937961 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:13.938704 17946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:13.939427 17938 server_base.cc:1034] running on GCE node
I20260929 01:04:13.940322 17938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:13.941442 17938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:13.942672 17938 hybrid_clock.cc:648] HybridClock initialized: now 1790643853942634 us; error 46 us; skew 500 ppm
I20260929 01:04:13.943215 17938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:13 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:13.947698 17938 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:04:13.949501 17938 webserver.cc:460] Webserver started at http://127.3.21.195:33865/ using document root <none> and password file <none>
I20260929 01:04:13.950415 17938 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:13.950644 17938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:13.956799 17938 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:04:13.960417 17953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:13.962450 17938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:13.962761 17938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "56a2a419e83c4936a27a0bf2057fc078"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "2f2fb53e218c957ca3ef8de6f200b5f2"
server_key_iv: "7b3fa6a57749c90c2080228408ad2d45"
server_key_version: "encryptionkey@0"
I20260929 01:04:13.963949 17938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:13.987217 17938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:13.992609 17938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:13.992929 17938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:13.994149 17938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:13.995764 17938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:13.995903 17938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:13.996044 17938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:13.996151 17938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:14.037218 17938 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37187
I20260929 01:04:14.037361 18065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37187 every 8 connection(s)
I20260929 01:04:14.038923 17938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:14.047562 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 17938
Sep 29 01:04:14 dist-test-slave-k0b7 krb5kdc[17028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:14.064937 17803 heartbeater.cc:503] Master 127.3.21.254:36387 was elected leader, sending a full tablet report...
I20260929 01:04:14.067225 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.042319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:37561 (local address 127.3.21.254:36387)
0929 01:04:14.042579 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.042588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.044529 (+ 1941us) server_negotiation.cc:407] Connection header received
0929 01:04:14.047287 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.047293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.047401 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.047537 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:14.050376 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.051666 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.054326 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.055070 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.055226 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.059656 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:14.059661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:14.059664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:14.059726 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:14.063386 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.064323 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.064331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.064338 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.064522 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.065022 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.065029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.065032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.065396 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:14.065566 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.066613 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.066856 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":6016}
I20260929 01:04:14.068133 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.048682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.3.21.254:36387)
0929 01:04:14.049779 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.049788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.049809 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:04:14.049869 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.049874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.049949 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.050107 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:14.052367 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.053732 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.055857 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.056468 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.056601 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.059656 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:14.059661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:14.059664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:14.059731 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:14.063277 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.064409 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.064413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.064415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.064554 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.065641 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.065644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.065647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.065989 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:14.066271 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.067501 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.067762 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"spinlock_wait_cycles":25600,"thread_start_us":782,"threads_started":1}
I20260929 01:04:14.068667 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.042734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:37561)
0929 01:04:14.044286 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:14.044387 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:14.046857 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:14.047882 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:14.047892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:14.048765 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:14.049972 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.050008 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.051904 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.051909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.054082 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.054097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.055366 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.055371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.055608 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.056563 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:14.056599 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:14.059370 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:14.063572 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:14.063581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:14.063598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:14.064112 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:14.064665 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:14.064670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:14.064674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:14.064895 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:14.065558 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:14.065567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:14.066075 (+ 508us) client_negotiation.cc:769] Sending connection context
0929 01:04:14.066464 (+ 389us) client_negotiation.cc:240] Negotiation successful
0929 01:04:14.067097 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":426,"threads_started":1}
I20260929 01:04:14.070432 18066 heartbeater.cc:347] Connected to a master server at 127.3.21.254:36387
I20260929 01:04:14.070818 18066 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:14.071885 18066 heartbeater.cc:511] Master 127.3.21.254:36387 requested a full tablet report, sending...
I20260929 01:04:14.073706 17617 ts_manager.cc:194] Registered new tserver with Master: 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187)
I20260929 01:04:14.074607 17617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:37561
I20260929 01:04:14.084740 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:14.093374 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.085960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.3.21.254:36387)
0929 01:04:14.086200 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.086205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.086682 (+ 477us) server_negotiation.cc:407] Connection header received
0929 01:04:14.086926 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.086932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.087120 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.087274 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:14.087764 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.088689 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.090384 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.090979 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.091182 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.091759 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:14.092724 (+ 965us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:14.092901 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.093054 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.093157 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:04:14.094619 17617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:04:14.102604 17617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:14.130681 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.116289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42599 (local address 127.0.0.1:41302)
0929 01:04:14.117824 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:14.117894 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:14.118140 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:14.121398 (+ 3258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:14.121411 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:14.121467 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:14.122066 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.122082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.124360 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.124368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.126942 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.126957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.129447 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.129454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.129648 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.129704 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 01:04:14.129784 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:04:14.130060 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":2,"thread_start_us":821,"threads_started":1}
I20260929 01:04:14.131844 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.116774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.3.21.193:42599)
0929 01:04:14.119789 (+ 3015us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.119799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.119829 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:04:14.119939 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.119949 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.120913 (+ 964us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.122095 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:14.122400 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.124208 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.127150 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.129159 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.130913 (+ 1754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.131176 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.131268 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.131467 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2749,"thread_start_us":2622,"threads_started":1}
I20260929 01:04:14.132339 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.117362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.3.21.194:32955)
0929 01:04:14.119162 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.119173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.121668 (+ 2495us) server_negotiation.cc:407] Connection header received
0929 01:04:14.121790 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.121798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.122107 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.122310 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:14.123249 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.124810 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.127401 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.129041 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.129269 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.131540 (+ 2271us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.131620 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.131753 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1484,"thread_start_us":1332,"threads_started":1}
I20260929 01:04:14.132542 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.119006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.3.21.195:37187)
0929 01:04:14.124050 (+ 5044us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.124060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.124092 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:04:14.124201 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.124210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.124470 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.124678 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:14.125472 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.127732 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.129470 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.131020 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.131155 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.131367 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.132007 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.132187 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4852,"thread_start_us":4692,"threads_started":1}
I20260929 01:04:14.132578 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.116758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:32955 (local address 127.0.0.1:47034)
0929 01:04:14.121361 (+ 4603us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:14.121382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:14.121508 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:14.122616 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:14.122621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:14.122647 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:14.123013 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.123025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.124992 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.125003 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.127222 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.127237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.129322 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.129331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.129569 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.129632 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 01:04:14.129751 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 01:04:14.132209 (+ 2458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4429,"mutex_wait_us":2242,"thread_start_us":3753,"threads_started":1}
I20260929 01:04:14.133313 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.118258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37187 (local address 127.0.0.1:43232)
0929 01:04:14.121625 (+ 3367us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:14.121650 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:14.121828 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:14.124688 (+ 2860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:14.124695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:14.124728 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:14.125231 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.125249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.127877 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.127885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.129261 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.129277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.131517 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.131523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.131713 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.131747 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:04:14.131857 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 01:04:14.132982 (+ 1125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3194,"thread_start_us":3952,"threads_started":1}
I20260929 01:04:14.134199 17738 tablet_service.cc:1467] Processing CreateTablet for tablet d4478cef507b4671813b45f631724047 (DEFAULT_TABLE table=test-table [id=ca0b04ad31354bd486171647de1a204f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:14.135828 17738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4478cef507b4671813b45f631724047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:14.136566 17870 tablet_service.cc:1467] Processing CreateTablet for tablet d4478cef507b4671813b45f631724047 (DEFAULT_TABLE table=test-table [id=ca0b04ad31354bd486171647de1a204f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:14.137066 18001 tablet_service.cc:1467] Processing CreateTablet for tablet d4478cef507b4671813b45f631724047 (DEFAULT_TABLE table=test-table [id=ca0b04ad31354bd486171647de1a204f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:14.138095 17870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4478cef507b4671813b45f631724047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:14.138399 18001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4478cef507b4671813b45f631724047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:14.145051 18082 tablet_bootstrap.cc:492] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Bootstrap starting.
I20260929 01:04:14.151036 18082 tablet_bootstrap.cc:654] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:14.152187 18082 log.cc:824] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:14.155763 18082 tablet_bootstrap.cc:492] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: No bootstrap required, opened a new log
I20260929 01:04:14.156121 18082 ts_tablet_manager.cc:1397] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260929 01:04:14.157416 18084 tablet_bootstrap.cc:492] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Bootstrap starting.
I20260929 01:04:14.159101 18085 tablet_bootstrap.cc:492] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Bootstrap starting.
I20260929 01:04:14.161942 18084 tablet_bootstrap.cc:654] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:14.162259 18085 tablet_bootstrap.cc:654] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:14.163436 18085 log.cc:824] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:14.163468 18084 log.cc:824] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:14.167138 18085 tablet_bootstrap.cc:492] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: No bootstrap required, opened a new log
I20260929 01:04:14.167472 18085 ts_tablet_manager.cc:1397] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 01:04:14.168829 18084 tablet_bootstrap.cc:492] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: No bootstrap required, opened a new log
I20260929 01:04:14.168624 18082 raft_consensus.cc:348] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.169122 18084 ts_tablet_manager.cc:1397] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 01:04:14.169142 18082 raft_consensus.cc:374] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:14.169239 18082 raft_consensus.cc:729] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784644fbadd24dd186195d9aecd1506b, State: Initialized, Role: FOLLOWER
I20260929 01:04:14.169946 18082 consensus_queue.cc:260] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.171980 18082 ts_tablet_manager.cc:1428] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260929 01:04:14.177893 18085 raft_consensus.cc:348] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.178355 18085 raft_consensus.cc:374] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:14.178486 18085 raft_consensus.cc:729] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f395dbcd8814cd892a14a44dad4fd16, State: Initialized, Role: FOLLOWER
I20260929 01:04:14.179217 18085 consensus_queue.cc:260] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.180105 18084 raft_consensus.cc:348] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.180521 17935 heartbeater.cc:503] Master 127.3.21.254:36387 was elected leader, sending a full tablet report...
I20260929 01:04:14.180617 18084 raft_consensus.cc:374] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:14.180720 18084 raft_consensus.cc:729] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a2a419e83c4936a27a0bf2057fc078, State: Initialized, Role: FOLLOWER
I20260929 01:04:14.181354 18085 ts_tablet_manager.cc:1428] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Time spent starting tablet: real 0.014s user 0.002s sys 0.010s
I20260929 01:04:14.181393 18084 consensus_queue.cc:260] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.182785 18066 heartbeater.cc:503] Master 127.3.21.254:36387 was elected leader, sending a full tablet report...
I20260929 01:04:14.183564 18084 ts_tablet_manager.cc:1428] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 01:04:14.193403 18088 raft_consensus.cc:482] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:14.193753 18088 raft_consensus.cc:504] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.195775 18088 leader_election.cc:288] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194:32955), 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187)
I20260929 01:04:14.205963 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.196412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:46805 (local address 127.3.21.195:37187)
0929 01:04:14.196624 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.196630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.197415 (+ 785us) server_negotiation.cc:407] Connection header received
0929 01:04:14.197601 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.197606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.197688 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.197798 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:14.198778 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.200036 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.203308 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.204716 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.204910 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.204992 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.205529 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.205721 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 01:04:14.205955 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.196307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:53485 (local address 127.3.21.194:32955)
0929 01:04:14.196553 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.196562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.197480 (+ 918us) server_negotiation.cc:407] Connection header received
0929 01:04:14.197598 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.197604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.197688 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.197795 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:14.198778 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.200029 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.203305 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.204716 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.204910 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.204991 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.205529 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.205677 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:04:14.206326 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.196277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:32955 (local address 127.3.21.193:53485)
0929 01:04:14.197266 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:14.197288 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:14.197455 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:14.197857 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:14.197865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:14.197914 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:14.198536 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.198553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.200218 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.200225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.203091 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.203115 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.205030 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.205037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.205276 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.205360 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 01:04:14.205505 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 01:04:14.205644 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":289,"spinlock_wait_cycles":14976,"thread_start_us":526,"threads_started":1}
I20260929 01:04:14.206598 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.196280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:37187 (local address 127.3.21.193:46805)
0929 01:04:14.197273 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:14.197288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:14.197455 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:14.197857 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:14.197865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:14.197916 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:14.198579 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.198589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.200218 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.200224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.203091 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:14.203115 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.205042 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.205047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:14.205345 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.205370 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:04:14.205469 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:04:14.205660 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":299,"spinlock_wait_cycles":80512,"thread_start_us":538,"threads_started":1}
I20260929 01:04:14.207768 18021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4478cef507b4671813b45f631724047" candidate_uuid: "784644fbadd24dd186195d9aecd1506b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a2a419e83c4936a27a0bf2057fc078" is_pre_election: true
I20260929 01:04:14.207768 17890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4478cef507b4671813b45f631724047" candidate_uuid: "784644fbadd24dd186195d9aecd1506b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f395dbcd8814cd892a14a44dad4fd16" is_pre_election: true
I20260929 01:04:14.208380 17890 raft_consensus.cc:2454] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 784644fbadd24dd186195d9aecd1506b in term 0.
I20260929 01:04:14.208400 18021 raft_consensus.cc:2454] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 784644fbadd24dd186195d9aecd1506b in term 0.
I20260929 01:04:14.209632 17690 leader_election.cc:302] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f395dbcd8814cd892a14a44dad4fd16, 784644fbadd24dd186195d9aecd1506b; no voters:
I20260929 01:04:14.210232 18088 raft_consensus.cc:2781] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:14.210405 18088 raft_consensus.cc:482] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:14.210551 18088 raft_consensus.cc:3037] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:14.212899 18088 raft_consensus.cc:504] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.213591 18088 leader_election.cc:288] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [CANDIDATE]: Term 1 election: Requested vote from peers 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194:32955), 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187)
I20260929 01:04:14.214236 17890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4478cef507b4671813b45f631724047" candidate_uuid: "784644fbadd24dd186195d9aecd1506b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f395dbcd8814cd892a14a44dad4fd16"
I20260929 01:04:14.214514 17890 raft_consensus.cc:3037] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:14.214465 18021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4478cef507b4671813b45f631724047" candidate_uuid: "784644fbadd24dd186195d9aecd1506b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a2a419e83c4936a27a0bf2057fc078"
I20260929 01:04:14.214866 18021 raft_consensus.cc:3037] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:14.217173 18021 raft_consensus.cc:2454] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 784644fbadd24dd186195d9aecd1506b in term 1.
I20260929 01:04:14.217180 17890 raft_consensus.cc:2454] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 784644fbadd24dd186195d9aecd1506b in term 1.
I20260929 01:04:14.217737 17691 leader_election.cc:302] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56a2a419e83c4936a27a0bf2057fc078, 784644fbadd24dd186195d9aecd1506b; no voters:
I20260929 01:04:14.218232 18088 raft_consensus.cc:2781] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:14.218771 18088 raft_consensus.cc:686] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 1 LEADER]: Becoming Leader. State: Replica: 784644fbadd24dd186195d9aecd1506b, State: Running, Role: LEADER
I20260929 01:04:14.219302 18088 consensus_queue.cc:237] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } }
I20260929 01:04:14.224599 17616 catalog_manager.cc:5495] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b reported cstate change: term changed from 0 to 1, leader changed from <none> to 784644fbadd24dd186195d9aecd1506b (127.3.21.193). New cstate: current_term: 1 leader_uuid: "784644fbadd24dd186195d9aecd1506b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "784644fbadd24dd186195d9aecd1506b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 } health_report { overall_health: UNKNOWN } } }
I20260929 01:04:14.268811 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.262301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.3.21.193:42599)
0929 01:04:14.262593 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.262597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.262683 (+ 86us) server_negotiation.cc:407] Connection header received
0929 01:04:14.262874 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.262877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.262971 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.263105 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:14.263532 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.264290 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.265791 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.266288 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.266597 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.267259 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:14.267905 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:14.268059 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.268251 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.268450 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 01:04:14.274610 17804 tablet.cc:2369] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:14.278697 18021 raft_consensus.cc:1261] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Refusing update from remote peer 784644fbadd24dd186195d9aecd1506b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:14.278697 17890 raft_consensus.cc:1261] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Refusing update from remote peer 784644fbadd24dd186195d9aecd1506b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:14.279981 18093 consensus_queue.cc:1035] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f395dbcd8814cd892a14a44dad4fd16" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:14.280517 18088 consensus_queue.cc:1035] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a2a419e83c4936a27a0bf2057fc078" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:04:14.289036 17936 tablet.cc:2369] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:14.290885 18067 tablet.cc:2369] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:14.295506 18096 mvcc.cc:205] Tried to move back new op lower bound from 7334477227112288256 to 7334477226885959680. Current Snapshot: MvccSnapshot[applied={T|T < 7334477227112288256}]
I20260929 01:04:14.317198 17617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:14.317500 17617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:14.321863 17617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Sending DeleteTablet for 3 replicas of tablet d4478cef507b4671813b45f631724047
I20260929 01:04:14.323314 17738 tablet_service.cc:1514] Processing DeleteTablet for tablet d4478cef507b4671813b45f631724047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:14 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:41302
I20260929 01:04:14.323334 17870 tablet_service.cc:1514] Processing DeleteTablet for tablet d4478cef507b4671813b45f631724047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:14 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:47034
I20260929 01:04:14.323628 18001 tablet_service.cc:1514] Processing DeleteTablet for tablet d4478cef507b4671813b45f631724047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:14 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:43232
I20260929 01:04:14.324363 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17654
I20260929 01:04:14.339181 18107 tablet_replica.cc:330] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: stopping tablet replica
I20260929 01:04:14.339903 18107 raft_consensus.cc:2229] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:04:14.340839 18107 raft_consensus.cc:2258] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:14.343622 18107 ts_tablet_manager.cc:1905] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:14.350273 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.326305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.3.21.193:42599)
0929 01:04:14.326580 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.326590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.326615 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:04:14.326689 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.326696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.326811 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.326936 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:14.329634 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.331376 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.334209 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.335171 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.335340 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.340437 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:14.340450 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:14.340468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:14.340521 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:14.345104 (+ 4583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.346396 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.346406 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.346409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.346652 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.347246 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.347252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.347256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.347760 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:14.348104 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.349525 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.349846 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:04:14.351213 17798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:14.351613 18108 tablet_replica.cc:330] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: stopping tablet replica
I20260929 01:04:14.352154 18108 raft_consensus.cc:2229] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:14.352540 18109 tablet_replica.cc:330] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: stopping tablet replica
I20260929 01:04:14.352792 18108 raft_consensus.cc:2258] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:14.353016 18109 raft_consensus.cc:2229] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:14.353737 18109 raft_consensus.cc:2258] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:14.356051 18109 ts_tablet_manager.cc:1905] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:14.356220 18108 ts_tablet_manager.cc:1905] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:14.362010 18109 ts_tablet_manager.cc:1918] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:14.362226 18109 log.cc:1196] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/d4478cef507b4671813b45f631724047
I20260929 01:04:14.362874 18109 ts_tablet_manager.cc:1939] T d4478cef507b4671813b45f631724047 P 56a2a419e83c4936a27a0bf2057fc078: Deleting consensus metadata
I20260929 01:04:14.364705 17602 catalog_manager.cc:4841] TS 56a2a419e83c4936a27a0bf2057fc078 (127.3.21.195:37187): tablet d4478cef507b4671813b45f631724047 (table test-table [id=ca0b04ad31354bd486171647de1a204f]) successfully deleted
I20260929 01:04:14.365496 18108 ts_tablet_manager.cc:1918] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:14.365708 18108 log.cc:1196] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/d4478cef507b4671813b45f631724047
I20260929 01:04:14.366354 18108 ts_tablet_manager.cc:1939] T d4478cef507b4671813b45f631724047 P 0f395dbcd8814cd892a14a44dad4fd16: Deleting consensus metadata
I20260929 01:04:14.368716 17601 catalog_manager.cc:4841] TS 0f395dbcd8814cd892a14a44dad4fd16 (127.3.21.194:32955): tablet d4478cef507b4671813b45f631724047 (table test-table [id=ca0b04ad31354bd486171647de1a204f]) successfully deleted
I20260929 01:04:14.457036 18107 ts_tablet_manager.cc:1918] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:14.457331 18107 log.cc:1196] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/d4478cef507b4671813b45f631724047
I20260929 01:04:14.457773 18107 ts_tablet_manager.cc:1939] T d4478cef507b4671813b45f631724047 P 784644fbadd24dd186195d9aecd1506b: Deleting consensus metadata
I20260929 01:04:14.459668 17602 catalog_manager.cc:4841] TS 784644fbadd24dd186195d9aecd1506b (127.3.21.193:42599): tablet d4478cef507b4671813b45f631724047 (table test-table [id=ca0b04ad31354bd486171647de1a204f]) successfully deleted
I20260929 01:04:14.463411 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17654
I20260929 01:04:14.484071 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17807
I20260929 01:04:14.499653 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.485015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.3.21.194:32955)
0929 01:04:14.485217 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.485223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.485246 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:04:14.485355 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.485361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.485450 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.485562 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:14.487258 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.488098 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.489553 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.489976 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.490120 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.493101 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:14.493118 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:14.493137 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:14.493172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:14.496476 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.497338 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.497346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.497349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.497489 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.497876 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.497882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.497886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.498263 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:14.498596 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.499057 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.499291 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:04:14.500527 17930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:14.594384 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17807
I20260929 01:04:14.612078 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17938
I20260929 01:04:14.628288 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:14.613064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.3.21.195:37187)
0929 01:04:14.613287 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:14.613293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:14.613318 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:04:14.613375 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:14.613379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:14.613455 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:14.613578 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:14.615195 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.615965 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.617757 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:14.618269 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:14.618386 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:14.621445 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:14.621461 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:14.621479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:14.621523 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:14.624740 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.625595 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.625604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.625606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.625766 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:14.626250 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:14.626255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:14.626258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:14.627018 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:14.627389 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:14.627711 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 01:04:14.627922 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:04:14.629058 18061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:14.727404 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17938
I20260929 01:04:14.745752 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 17584
I20260929 01:04:14.747392 17647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:14.873929 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 17584
I20260929 01:04:14.897765 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36387
--webserver_interface=127.3.21.254
--webserver_port=34419
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:15.256711 18114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:15.257109 18114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:15.257270 18114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:15.265214 18114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:15.265367 18114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:15.265465 18114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:15.265573 18114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:15.265662 18114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:15.276291 18114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:15.277812 18114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:15.279212 18114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:15.292034 18119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:15.292214 18120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:15.292217 18122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:15.293156 18114 server_base.cc:1034] running on GCE node
I20260929 01:04:15.294054 18114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:15.295782 18114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:15.297076 18114 hybrid_clock.cc:648] HybridClock initialized: now 1790643855297004 us; error 45 us; skew 500 ppm
I20260929 01:04:15.297465 18114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:15 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:15.309227 18114 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:15.311245 18114 webserver.cc:460] Webserver started at http://127.3.21.254:34419/ using document root <none> and password file <none>
I20260929 01:04:15.312013 18114 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:15.312132 18114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:15.318673 18114 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 01:04:15.324421 18129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:15.325930 18114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:04:15.326136 18114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "45461be729e04161b94e81c9855d0b4c"
format_stamp: "Formatted at 2026-09-29 01:04:08 on dist-test-slave-k0b7"
server_key: "b17adf962b9ca9eec0c1c22f2b89dbc9"
server_key_iv: "7d95cb131b84ce1036fd21a9a969ddf6"
server_key_version: "encryptionkey@0"
I20260929 01:04:15.327332 18114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:15.347110 18114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:15.352562 18114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:15.352933 18114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:15.375639 18114 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36387
I20260929 01:04:15.375702 18180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36387 every 8 connection(s)
I20260929 01:04:15.377326 18114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:15.385545 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18114
I20260929 01:04:15.385931 18181 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:04:15.386656 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:42599
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:36387
--webserver_port=42471
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:15.392292 18181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap starting.
I20260929 01:04:15.415460 18181 log.cc:824] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:15.435969 18181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:15.436666 18181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c: Bootstrap complete.
I20260929 01:04:15.447818 18181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:15.448905 18181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45461be729e04161b94e81c9855d0b4c, State: Initialized, Role: FOLLOWER
I20260929 01:04:15.449613 18181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:15.449878 18181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:15.450024 18181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:15.450245 18181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:04:15.455204 18181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:15.455807 18181 leader_election.cc:302] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45461be729e04161b94e81c9855d0b4c; no voters:
I20260929 01:04:15.456589 18181 leader_election.cc:288] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:04:15.456756 18185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:04:15.457965 18185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [term 3 LEADER]: Becoming Leader. State: Replica: 45461be729e04161b94e81c9855d0b4c, State: Running, Role: LEADER
I20260929 01:04:15.458626 18185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } }
I20260929 01:04:15.459226 18181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:15.462334 18186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "45461be729e04161b94e81c9855d0b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } } }
I20260929 01:04:15.462911 18186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:15.462280 18187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45461be729e04161b94e81c9855d0b4c. Latest consensus state: current_term: 3 leader_uuid: "45461be729e04161b94e81c9855d0b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45461be729e04161b94e81c9855d0b4c" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36387 } } }
I20260929 01:04:15.463116 18187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45461be729e04161b94e81c9855d0b4c [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:15.476362 18193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:15.486183 18193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:15.488343 18193 catalog_manager.cc:1259] Loaded cluster ID: f84e398fd0624ddaadccbb840876b2ee
I20260929 01:04:15.488480 18193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 01:04:15.491492 18193 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd7973999ad google::LogMessage::Fail() at ??:0
@ 0x7fd79739c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd7973994de google::LogMessage::Flush() at ??:0
@ 0x7fd79739aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd7acdcc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd7acdc9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd7ace51610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd7ace51439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd799fe5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd799fe891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd799fe8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd799fc3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd79d29d6db start_thread at ??:0
@ 0x7fd79623361f clone at ??:0
W20260929 01:04:15.830756 18183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:15.831146 18183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:15.831358 18183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:15.840006 18183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:15.840163 18183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:15.840380 18183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:04:15.852046 18183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=42471
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:15.853506 18183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:15.855252 18183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:15.870114 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:15.870160 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:15.870136 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:15.871444 18183 server_base.cc:1034] running on GCE node
I20260929 01:04:15.872218 18183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:15.873288 18183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:15.874620 18183 hybrid_clock.cc:648] HybridClock initialized: now 1790643855874458 us; error 42 us; skew 500 ppm
I20260929 01:04:15.874928 18183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:15 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:15.878330 18183 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:04:15.880123 18183 webserver.cc:460] Webserver started at http://127.3.21.193:42471/ using document root <none> and password file <none>
I20260929 01:04:15.880851 18183 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:15.880959 18183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:15.886869 18183 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:04:15.890123 18232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:15.891875 18183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:15.892148 18183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "784644fbadd24dd186195d9aecd1506b"
format_stamp: "Formatted at 2026-09-29 01:04:09 on dist-test-slave-k0b7"
server_key: "fe3b6e6eaf4b117419a1001c8ecc0ce2"
server_key_iv: "408a5109d83be7ea28446c6ce719f109"
server_key_version: "encryptionkey@0"
I20260929 01:04:15.893218 18183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:15.914463 18183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:15.919931 18183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:15.920217 18183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:15.921438 18183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:15.923197 18183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:15.923307 18183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:15.923398 18183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:15.923471 18183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:15.963225 18183 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:42599
I20260929 01:04:15.963325 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:42599 every 8 connection(s)
I20260929 01:04:15.964884 18183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:15.967839 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18183
I20260929 01:04:15.968495 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:32955
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:36387
--webserver_port=36581
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:15.970985 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:15.968419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:57203)
0929 01:04:15.969375 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:15.969696 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":371,"threads_started":1}
W20260929 01:04:15.971827 18345 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:36387 (0 consecutive failures): Network error: Failed to ping master at 127.3.21.254:36387: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
I20260929 01:04:15.973141 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:15.972687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:54767)
0929 01:04:15.972866 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:15.973019 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 01:04:15.974349 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:15.973884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:53785)
0929 01:04:15.974126 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:15.974211 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260929 01:04:15.974727 18345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:04:16.323918 18348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:16.324223 18348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:16.324400 18348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:16.332124 18348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:16.332276 18348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:16.332471 18348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:04:16.342700 18348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:32955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=36581
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:16.344157 18348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:16.345573 18348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:16.360496 18353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:16.360523 18356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:16.360646 18354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:16.361297 18348 server_base.cc:1034] running on GCE node
I20260929 01:04:16.362061 18348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:16.363181 18348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:16.364451 18348 hybrid_clock.cc:648] HybridClock initialized: now 1790643856364405 us; error 41 us; skew 500 ppm
I20260929 01:04:16.364856 18348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:16 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:16.368392 18348 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:04:16.369962 18348 webserver.cc:460] Webserver started at http://127.3.21.194:36581/ using document root <none> and password file <none>
I20260929 01:04:16.370844 18348 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:16.370993 18348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:16.377429 18348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:04:16.380826 18363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:16.382473 18348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:04:16.382865 18348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0f395dbcd8814cd892a14a44dad4fd16"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "bc89092b151124915a4cc3ba8ec17388"
server_key_iv: "72a3ec1b93a1f2364f4344de40e8a454"
server_key_version: "encryptionkey@0"
I20260929 01:04:16.383967 18348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:16.409328 18348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:16.414261 18348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:16.414832 18348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:16.416033 18348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:16.417555 18348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:16.417696 18348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:16.417842 18348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:16.417935 18348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:16.455564 18348 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:32955
I20260929 01:04:16.455646 18475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:32955 every 8 connection(s)
I20260929 01:04:16.457238 18348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:16.462688 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.460664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:42491)
0929 01:04:16.461447 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.461684 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":315,"threads_started":1}
W20260929 01:04:16.463450 18476 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:36387 (0 consecutive failures): Network error: Failed to ping master at 127.3.21.254:36387: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
I20260929 01:04:16.464563 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.464071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:52581)
0929 01:04:16.464312 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.464405 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 01:04:16.465461 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.465158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:43123)
0929 01:04:16.465282 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.465354 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 01:04:16.465726 18476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:04:16.467139 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18348
I20260929 01:04:16.467788 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:37187
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:36387
--webserver_port=33865
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:16.828994 18479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:16.829329 18479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:16.829432 18479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:16.837105 18479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:16.837258 18479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:16.837379 18479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:04:16.848933 18479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:37187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=33865
--tserver_master_addrs=127.3.21.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:16.850346 18479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:16.851701 18479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:16.866353 18484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:16.866245 18485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:16.866413 18487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:16.867386 18479 server_base.cc:1034] running on GCE node
I20260929 01:04:16.868152 18479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:16.869297 18479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:16.870608 18479 hybrid_clock.cc:648] HybridClock initialized: now 1790643856870563 us; error 36 us; skew 500 ppm
I20260929 01:04:16.870923 18479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:16 dist-test-slave-k0b7 krb5kdc[17028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:16.874709 18479 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:04:16.876451 18479 webserver.cc:460] Webserver started at http://127.3.21.195:33865/ using document root <none> and password file <none>
I20260929 01:04:16.877190 18479 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:16.877300 18479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:16.883832 18479 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:16.887326 18494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:16.889236 18479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:16.889423 18479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "56a2a419e83c4936a27a0bf2057fc078"
format_stamp: "Formatted at 2026-09-29 01:04:10 on dist-test-slave-k0b7"
server_key: "2f2fb53e218c957ca3ef8de6f200b5f2"
server_key_iv: "7b3fa6a57749c90c2080228408ad2d45"
server_key_version: "encryptionkey@0"
I20260929 01:04:16.890719 18479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:16.904906 18479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:16.910077 18479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:16.910435 18479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:16.911661 18479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:16.913282 18479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:16.913381 18479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:16.913556 18479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:16.913640 18479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:16.954653 18479 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:37187
I20260929 01:04:16.954821 18606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:37187 every 8 connection(s)
I20260929 01:04:16.956329 18479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:16.961488 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.959577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:34357)
0929 01:04:16.960339 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.960552 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":291,"threads_started":1}
W20260929 01:04:16.962735 18607 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:36387 (0 consecutive failures): Network error: Failed to ping master at 127.3.21.254:36387: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
I20260929 01:04:16.963863 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.963466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:39477)
0929 01:04:16.963658 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.963741 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:04:16.964380 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18479
I20260929 01:04:16.965202 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.964713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:47957)
0929 01:04:16.964901 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.965012 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 01:04:16.966204 18607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:04:16.976387 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:16.975506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:45243)
0929 01:04:16.976000 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:16.976134 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":239,"threads_started":1}
I20260929 01:04:17.467592 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:17.466633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:38229)
0929 01:04:17.467176 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:17.467280 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260929 01:04:17.967772 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:17.966930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:56445)
0929 01:04:17.967416 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:17.967499 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260929 01:04:17.978435 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:17.977505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:38863)
0929 01:04:17.978156 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:17.978258 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":303,"threads_started":1}
I20260929 01:04:18.469581 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:18.468797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:48985)
0929 01:04:18.469303 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:18.469393 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
I20260929 01:04:18.970013 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:18.969009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:53117)
0929 01:04:18.969683 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:18.969805 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":318,"threads_started":1}
I20260929 01:04:18.980659 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:18.979725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:58901)
0929 01:04:18.980358 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:18.980460 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":385,"threads_started":1}
I20260929 01:04:19.471472 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:19.470656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:57795)
0929 01:04:19.471183 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:19.471278 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":302,"threads_started":1}
I20260929 01:04:19.972193 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:19.971270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:53693)
0929 01:04:19.971794 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:19.971938 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260929 01:04:19.982856 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:19.981884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:41709)
0929 01:04:19.982467 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:19.982665 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":331,"threads_started":1}
I20260929 01:04:20.473598 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:20.472763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:54259)
0929 01:04:20.473302 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:20.473407 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":268,"threads_started":1}
I20260929 01:04:20.974143 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:20.973296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:49501)
0929 01:04:20.973846 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:20.973959 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":298,"threads_started":1}
I20260929 01:04:20.985222 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:20.984329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:58545)
0929 01:04:20.984834 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:20.984922 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
I20260929 01:04:21.476109 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:21.474954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:60413)
0929 01:04:21.475738 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:21.475863 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":393,"threads_started":1}
I20260929 01:04:21.976351 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:21.975487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:49193)
0929 01:04:21.976027 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:21.976128 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":287,"threads_started":1}
I20260929 01:04:21.987093 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:21.986252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:58627)
0929 01:04:21.986832 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:21.986925 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":338,"threads_started":1}
I20260929 01:04:22.478667 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:22.477591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:58299)
0929 01:04:22.478261 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:22.478378 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":385,"threads_started":1}
I20260929 01:04:22.978340 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:22.977521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:33785)
0929 01:04:22.978045 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:22.978154 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
I20260929 01:04:22.989132 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:22.988219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:32901)
0929 01:04:22.988836 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:22.988922 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":303,"threads_started":1}
I20260929 01:04:23.480856 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:23.479800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:44999)
0929 01:04:23.480461 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:23.480580 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":296,"threads_started":1}
I20260929 01:04:23.980424 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:23.979576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:54087)
0929 01:04:23.980109 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:23.980199 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":235,"threads_started":1}
I20260929 01:04:23.991204 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:23.990365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:40789)
0929 01:04:23.990946 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:23.991034 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":322,"threads_started":1}
I20260929 01:04:24.482954 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:24.482123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:37617)
0929 01:04:24.482691 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:24.482784 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":222,"threads_started":1}
I20260929 01:04:24.982724 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:24.981692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:54053)
0929 01:04:24.982225 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:24.982341 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":244,"threads_started":1}
I20260929 01:04:24.993314 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:24.992416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:35689)
0929 01:04:24.993027 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:24.993115 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":340,"threads_started":1}
I20260929 01:04:25.484917 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:25.484120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:56973)
0929 01:04:25.484634 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:25.484738 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":255,"threads_started":1}
I20260929 01:04:25.984779 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:25.983836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:37541)
0929 01:04:25.984465 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:25.984583 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":299,"threads_started":1}
I20260929 01:04:25.995107 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:25.994326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:51933)
0929 01:04:25.994863 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:25.994944 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":334,"threads_started":1}
I20260929 01:04:26.486917 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:26.486127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:34325)
0929 01:04:26.486658 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:26.486749 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":238,"threads_started":1}
I20260929 01:04:26.987301 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:26.986058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:33685)
0929 01:04:26.986877 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:26.986990 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":486,"threads_started":1}
I20260929 01:04:26.997514 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:26.996373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:60701)
0929 01:04:26.996903 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:26.997215 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260929 01:04:27.489344 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:27.488336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:37465)
0929 01:04:27.488897 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:27.489007 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":237,"threads_started":1}
I20260929 01:04:27.989295 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:27.988415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:35863)
0929 01:04:27.988979 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:27.989079 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
I20260929 01:04:27.999606 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:27.998824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:46905)
0929 01:04:27.999357 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:27.999447 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
I20260929 01:04:28.492034 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:28.490878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:33897)
0929 01:04:28.491567 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:28.491785 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":351,"threads_started":1}
I20260929 01:04:28.991539 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:28.990544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:36263)
0929 01:04:28.991274 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:28.991376 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":345,"threads_started":1}
I20260929 01:04:29.001828 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:29.000826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:41289)
0929 01:04:29.001517 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:29.001625 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":490,"threads_started":1}
I20260929 01:04:29.494261 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:29.493127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:37767)
0929 01:04:29.493811 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:29.493933 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":408,"threads_started":1}
I20260929 01:04:29.993826 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:29.992851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:42083)
0929 01:04:29.993494 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:29.993628 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":365,"threads_started":1}
I20260929 01:04:30.004149 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:30.003151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:38493)
0929 01:04:30.003850 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:30.003946 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":449,"threads_started":1}
I20260929 01:04:30.496320 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:30.495208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:59031)
0929 01:04:30.495871 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:30.495999 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":244,"threads_started":1}
I20260929 01:04:30.996012 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:30.995017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:37555)
0929 01:04:30.995666 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:30.995763 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":376,"threads_started":1}
I20260929 01:04:31.006717 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:31.005630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.193:37681)
0929 01:04:31.006289 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:31.006393 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":325,"threads_started":1}
I20260929 01:04:31.498660 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:31.497485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.194:53891)
0929 01:04:31.498121 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:31.498240 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":343,"threads_started":1}
I20260929 01:04:31.971868 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 18183
I20260929 01:04:31.995123 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:31.973214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.3.21.193:42599)
0929 01:04:31.974190 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:31.974214 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:31.974278 (+ 64us) server_negotiation.cc:407] Connection header received
0929 01:04:31.976050 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:31.976061 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:31.976556 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:31.976893 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:31.978658 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:31.980033 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:31.981604 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:31.982428 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:31.982664 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:31.986827 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:31.986839 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:31.986860 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:31.986915 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:31.990936 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:31.991788 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:31.991796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:31.991804 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:31.992099 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:31.992760 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:31.992764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:31.992767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:31.993459 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:31.993688 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:31.994297 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 01:04:31.994800 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":600,"threads_started":1}
I20260929 01:04:31.996666 18340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:31.998590 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:31.997470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:36387 (local address 127.3.21.195:57631)
0929 01:04:31.998132 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:31.998279 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260929 01:04:32.085572 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 18183
I20260929 01:04:32.103986 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 18348
I20260929 01:04:32.124655 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:32.105008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.3.21.194:32955)
0929 01:04:32.105592 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:32.105608 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:32.105646 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:04:32.107506 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:32.107523 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:32.108026 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:32.108397 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:32.110027 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:32.111474 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:32.112844 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:32.113609 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:32.113898 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:32.116946 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:32.116957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:32.116976 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:32.117024 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:32.120681 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:32.121529 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:32.121537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:32.121545 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:32.121748 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:32.122329 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:32.122333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:32.122336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:32.123094 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:32.123375 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:32.123878 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 01:04:32.124349 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":271,"threads_started":1}
I20260929 01:04:32.126282 18471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:32.216303 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 18348
I20260929 01:04:32.234858 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 18479
I20260929 01:04:32.256280 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:32.235990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.3.21.195:37187)
0929 01:04:32.236715 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:32.236730 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:32.236773 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:04:32.238569 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:32.238580 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:32.239062 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:32.239396 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:32.241002 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:32.242439 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:32.244205 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:32.245019 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:32.245184 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:32.248490 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:32.248501 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:32.248521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:32.248584 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:32.252077 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:32.253078 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:32.253087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:32.253095 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:32.253306 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:32.253860 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:32.253864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:32.253866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:32.254624 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:32.254881 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:32.255433 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 01:04:32.255962 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":416,"threads_started":1}
I20260929 01:04:32.258145 18602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:32.346251 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 18479
2026-09-29T01:04:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25488 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:04:32 dist-test-slave-k0b7 krb5kdc[18665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:04:32 dist-test-slave-k0b7 krb5kdc[18665](info): set up 2 sockets
Sep 29 01:04:32 dist-test-slave-k0b7 krb5kdc[18665](info): commencing operation
krb5kdc: starting...
W20260929 01:04:33.495471 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:04:33 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:04:33.763684 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34335
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:34.125694 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:34.126027 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:34.126189 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:34.133916 18678 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:04:34.134042 18678 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:04:34.134081 18678 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:04:34.134164 18678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:34.134258 18678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:34.134388 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:34.134454 18678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:34.134545 18678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:34.145213 18678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:34.146678 18678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:34.148041 18678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:34.161983 18683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:34.161924 18686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:34.162014 18684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:34.161926 18678 server_base.cc:1034] running on GCE node
I20260929 01:04:34.163286 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:34.164904 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:34.166218 18678 hybrid_clock.cc:648] HybridClock initialized: now 1790643874166137 us; error 53 us; skew 500 ppm
I20260929 01:04:34.166822 18678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:34 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:34.170667 18678 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:34.172432 18678 webserver.cc:460] Webserver started at http://127.3.21.254:37739/ using document root <none> and password file <none>
I20260929 01:04:34.173198 18678 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:34.173303 18678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:34.173591 18678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:34.177106 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "b16e1d50066440d0b43b4bbc20adb428"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "4665013b9f6d6a448b98133fb029817f"
server_key_iv: "c538e737f011d32f51a505b174cc60c9"
server_key_version: "encryptionkey@0"
I20260929 01:04:34.177983 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "b16e1d50066440d0b43b4bbc20adb428"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "4665013b9f6d6a448b98133fb029817f"
server_key_iv: "c538e737f011d32f51a505b174cc60c9"
server_key_version: "encryptionkey@0"
I20260929 01:04:34.184260 18678 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 01:04:34.188715 18693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:34.190572 18678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:34.190874 18678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "b16e1d50066440d0b43b4bbc20adb428"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "4665013b9f6d6a448b98133fb029817f"
server_key_iv: "c538e737f011d32f51a505b174cc60c9"
server_key_version: "encryptionkey@0"
I20260929 01:04:34.191139 18678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:34.206836 18678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:34.212373 18678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:34.212716 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:34.233498 18678 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34335
I20260929 01:04:34.233515 18744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34335 every 8 connection(s)
I20260929 01:04:34.235137 18678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:34.239487 18745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:34.241806 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18678
I20260929 01:04:34.242084 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:04:34.243001 3159 external_mini_cluster.cc:1428] Setting key 6c4f2b11b547406ea1b239159a03ab55
I20260929 01:04:34.254329 18745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap starting.
I20260929 01:04:34.258793 18745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Neither blocks nor log segments found. Creating new log.
Sep 29 01:04:34 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:34.260174 18745 log.cc:824] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:34.263530 18745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: No bootstrap required, opened a new log
I20260929 01:04:34.269452 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:34.245404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.3.21.254:34335)
0929 01:04:34.246340 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:34.246356 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:34.246405 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:04:34.248326 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:34.248353 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:34.248791 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:34.249138 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:34.251311 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.252798 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.254799 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.255679 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.255839 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:34.261208 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:34.261220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:34.261239 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:34.261293 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:34.264729 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:34.265552 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:34.265559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:34.265566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:34.265752 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:34.266199 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:34.266203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:34.266205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:34.266926 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:34.267302 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:34.267771 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 01:04:34.268300 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":438,"threads_started":1}
I20260929 01:04:34.272123 18745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:34.272604 18745 raft_consensus.cc:374] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:34.272722 18745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16e1d50066440d0b43b4bbc20adb428, State: Initialized, Role: FOLLOWER
I20260929 01:04:34.273347 18745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:34.273561 18745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:34.273661 18745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:34.273792 18745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:34.276245 18745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:34.276839 18745 leader_election.cc:302] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16e1d50066440d0b43b4bbc20adb428; no voters:
I20260929 01:04:34.277544 18745 leader_election.cc:288] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:04:34.277640 18750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:34.278568 18750 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 LEADER]: Becoming Leader. State: Replica: b16e1d50066440d0b43b4bbc20adb428, State: Running, Role: LEADER
I20260929 01:04:34.279407 18750 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:34.279965 18745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:34.282637 18752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16e1d50066440d0b43b4bbc20adb428. Latest consensus state: current_term: 1 leader_uuid: "b16e1d50066440d0b43b4bbc20adb428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } } }
I20260929 01:04:34.282598 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b16e1d50066440d0b43b4bbc20adb428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } } }
I20260929 01:04:34.283070 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:34.283072 18752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:34.291769 18756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:34.297667 18756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:34.308382 18756 catalog_manager.cc:1347] Generated new cluster ID: 0cb71c8e2b364b869d72c54483012479
I20260929 01:04:34.308516 18756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:34.341400 18756 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:34.343003 18756 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:34.349254 18756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Generated new TSK 0
I20260929 01:04:34.350458 18756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:04:34.424090 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34335
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:34.787618 18771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:34.787940 18771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:34.788095 18771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:34.795838 18771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:34.795958 18771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:34.796170 18771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:04:34.806411 18771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:34.807893 18771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:34.809264 18771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:34.824340 18776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:34.824609 18779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:34.824369 18777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:34.825138 18771 server_base.cc:1034] running on GCE node
I20260929 01:04:34.825788 18771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:34.826901 18771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:34.828189 18771 hybrid_clock.cc:648] HybridClock initialized: now 1790643874828143 us; error 43 us; skew 500 ppm
I20260929 01:04:34.828637 18771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:34 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:34.839577 18771 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:04:34.841564 18771 webserver.cc:460] Webserver started at http://127.3.21.193:36461/ using document root <none> and password file <none>
I20260929 01:04:34.842402 18771 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:34.842615 18771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:34.843000 18771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:34.846570 18771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "abad13a5625b48bfbff8b5f2f96b5450"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "bfd7fdf9ba65d19ec546cf481e7cdeaf"
server_key_iv: "0de466ababf0490549d1ececd423ea21"
server_key_version: "encryptionkey@0"
I20260929 01:04:34.847577 18771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "abad13a5625b48bfbff8b5f2f96b5450"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "bfd7fdf9ba65d19ec546cf481e7cdeaf"
server_key_iv: "0de466ababf0490549d1ececd423ea21"
server_key_version: "encryptionkey@0"
I20260929 01:04:34.853801 18771 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 01:04:34.857972 18786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:34.859555 18771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:34.859771 18771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "abad13a5625b48bfbff8b5f2f96b5450"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "bfd7fdf9ba65d19ec546cf481e7cdeaf"
server_key_iv: "0de466ababf0490549d1ececd423ea21"
server_key_version: "encryptionkey@0"
I20260929 01:04:34.860018 18771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:34.894465 18771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:34.900138 18771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:34.900451 18771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:34.901471 18771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:34.903157 18771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:34.903275 18771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:34.903445 18771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:34.903537 18771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:34.943812 18771 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:43597
I20260929 01:04:34.943928 18898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:43597 every 8 connection(s)
I20260929 01:04:34.945616 18771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:34.954991 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18771
I20260929 01:04:34.955371 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:04:34.956373 3159 external_mini_cluster.cc:1428] Setting key 95fdd7d3904ffbb4ef6ce5623456f485
Sep 29 01:04:34 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:34.971513 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:34.948969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:38503 (local address 127.3.21.254:34335)
0929 01:04:34.949440 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:34.949445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:34.950612 (+ 1167us) server_negotiation.cc:407] Connection header received
0929 01:04:34.952695 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:34.952699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:34.952787 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:34.952896 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:34.954803 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.956142 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.958193 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.958998 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.959163 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:34.964339 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:34.964345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:34.964349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:34.964388 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:34.967481 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:34.968428 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:34.968435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:34.968443 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:34.968624 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:34.969140 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:34.969148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:34.969155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:34.969581 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:34.969770 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:34.970662 (+ 892us) server_negotiation.cc:299] Negotiation successful
0929 01:04:34.970971 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260929 01:04:34.972294 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:34.949564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:38503)
0929 01:04:34.950388 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:34.950429 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:34.952377 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:34.953188 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:34.953196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:34.953879 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:34.954555 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:34.954589 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.956404 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.956408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:34.958018 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:34.958031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.959342 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.959353 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:34.959742 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:34.960604 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:34.960698 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:34.964009 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:34.967712 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:34.967720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:34.967736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:34.968257 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:34.968768 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:34.968772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:34.968775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:34.969000 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:34.969740 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:34.969770 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:34.970155 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 01:04:34.970448 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 01:04:34.971026 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":333,"threads_started":1}
I20260929 01:04:34.974479 18899 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34335
I20260929 01:04:34.974877 18899 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:34.975894 18899 heartbeater.cc:511] Master 127.3.21.254:34335 requested a full tablet report, sending...
I20260929 01:04:34.978865 18710 ts_manager.cc:194] Registered new tserver with Master: abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193:43597)
I20260929 01:04:34.981527 18710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:38503
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:04:35.023702 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34335
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:35.375005 18905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:35.375335 18905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:35.375499 18905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:35.383083 18905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:35.383210 18905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:35.383399 18905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:04:35.394294 18905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:35.395823 18905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:35.397236 18905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:35.412254 18910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.412586 18913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.413384 18911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:35.413820 18905 server_base.cc:1034] running on GCE node
I20260929 01:04:35.414670 18905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:35.415828 18905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:35.417101 18905 hybrid_clock.cc:648] HybridClock initialized: now 1790643875417028 us; error 70 us; skew 500 ppm
I20260929 01:04:35.417536 18905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:35 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:35.421561 18905 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:04:35.423619 18905 webserver.cc:460] Webserver started at http://127.3.21.194:38229/ using document root <none> and password file <none>
I20260929 01:04:35.424404 18905 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:35.424566 18905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:35.424906 18905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:35.428134 18905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "0618370d00ae4c9087451377f65a7ff3"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "f53f50efa4a99cc81b621778b48d464a"
server_key_iv: "edd2a13becf6d66dc561185074299e7b"
server_key_version: "encryptionkey@0"
I20260929 01:04:35.428961 18905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "0618370d00ae4c9087451377f65a7ff3"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "f53f50efa4a99cc81b621778b48d464a"
server_key_iv: "edd2a13becf6d66dc561185074299e7b"
server_key_version: "encryptionkey@0"
I20260929 01:04:35.435215 18905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:04:35.440121 18920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.442135 18905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:04:35.442408 18905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0618370d00ae4c9087451377f65a7ff3"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "f53f50efa4a99cc81b621778b48d464a"
server_key_iv: "edd2a13becf6d66dc561185074299e7b"
server_key_version: "encryptionkey@0"
I20260929 01:04:35.442732 18905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:35.467988 18905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:35.472886 18905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:35.473207 18905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:35.474267 18905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:35.475742 18905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:35.475849 18905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.476001 18905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:35.476073 18905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.512209 18905 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:46499
I20260929 01:04:35.512353 19032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:46499 every 8 connection(s)
I20260929 01:04:35.513590 18905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:35.520591 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 18905
I20260929 01:04:35.520818 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:04:35.521466 3159 external_mini_cluster.cc:1428] Setting key df157ac58e83b6e231483d529ea76c60
Sep 29 01:04:35 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:35.538373 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:35.516266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:39885 (local address 127.3.21.254:34335)
0929 01:04:35.516764 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:35.516771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:35.517908 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 01:04:35.519989 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:35.519994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:35.520077 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:35.520189 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:35.522567 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.523715 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.526674 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.527219 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.527350 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:35.532146 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:35.532152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:35.532155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:35.532192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:35.535108 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:35.535945 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:35.535950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:35.535952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:35.536079 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:35.536563 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:35.536568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:35.536571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:35.536854 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:35.537007 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:35.537736 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 01:04:35.537968 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
I20260929 01:04:35.539593 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:35.516862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:39885)
0929 01:04:35.517713 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:35.517755 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:35.519669 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:35.520614 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:35.520630 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:35.521462 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:35.522282 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:35.522329 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.523963 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.523969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:35.526310 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:35.526344 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.527502 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.527511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:35.527883 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:35.528803 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:35.528837 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:35.531849 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:35.535282 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:35.535292 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:35.535313 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:35.535785 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:35.536217 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:35.536222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:35.536226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:35.536442 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:35.536995 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:35.537003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:35.537376 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 01:04:35.537660 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 01:04:35.538253 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":396,"threads_started":1}
I20260929 01:04:35.541272 19033 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34335
I20260929 01:04:35.541674 19033 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:35.542721 19033 heartbeater.cc:511] Master 127.3.21.254:34335 requested a full tablet report, sending...
I20260929 01:04:35.544956 18710 ts_manager.cc:194] Registered new tserver with Master: 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499)
I20260929 01:04:35.546423 18710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:39885
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:04:35.583509 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34335
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:35.931011 19039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:35.931370 19039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:35.931545 19039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:35.939515 19039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:35.939678 19039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:35.939870 19039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:04:35.950654 19039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:35.952019 19039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:35.953382 19039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:35.967808 19045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.967829 19047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.967906 19044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:35.968721 19039 server_base.cc:1034] running on GCE node
I20260929 01:04:35.969448 19039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:35.970443 19039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:35.971732 19039 hybrid_clock.cc:648] HybridClock initialized: now 1790643875971675 us; error 87 us; skew 500 ppm
I20260929 01:04:35.972079 19039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:35 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:35.975543 19039 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:04:35.977231 19039 webserver.cc:460] Webserver started at http://127.3.21.195:44817/ using document root <none> and password file <none>
I20260929 01:04:35.977991 19039 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:35.978154 19039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:35.978641 19039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:35.981953 19039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "44a8b4570e4f446691045bf93a05c015"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "4b5b0c08d435be63cc51ac550d3d595f"
server_key_iv: "78a87290acbff24e4fa770edd45425d9"
server_key_version: "encryptionkey@0"
I20260929 01:04:35.982903 19039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "44a8b4570e4f446691045bf93a05c015"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "4b5b0c08d435be63cc51ac550d3d595f"
server_key_iv: "78a87290acbff24e4fa770edd45425d9"
server_key_version: "encryptionkey@0"
I20260929 01:04:35.986795 18899 heartbeater.cc:503] Master 127.3.21.254:34335 was elected leader, sending a full tablet report...
I20260929 01:04:35.989157 19039 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260929 01:04:35.993546 19054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.995517 19039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:35.995779 19039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "44a8b4570e4f446691045bf93a05c015"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "4b5b0c08d435be63cc51ac550d3d595f"
server_key_iv: "78a87290acbff24e4fa770edd45425d9"
server_key_version: "encryptionkey@0"
I20260929 01:04:35.996016 19039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:36.012444 19039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:36.018620 19039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:36.019025 19039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:36.020259 19039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:36.021894 19039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:36.022017 19039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.022202 19039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:36.022289 19039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.063292 19039 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:41277
I20260929 01:04:36.063426 19166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:41277 every 8 connection(s)
I20260929 01:04:36.064848 19039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:36.070657 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19039
I20260929 01:04:36.070875 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:04:36.071496 3159 external_mini_cluster.cc:1428] Setting key 61712622fe1f9449e67b867f27177375
Sep 29 01:04:36 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:36.089581 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.067954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:34513 (local address 127.3.21.254:34335)
0929 01:04:36.068422 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.068426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.069530 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 01:04:36.072334 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.072340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.072458 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.072589 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.074531 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.075354 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.077025 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.077459 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.077606 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.082011 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.082017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.082021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.082059 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.085405 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.086539 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.086544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.086548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.086754 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.087342 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.087347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.087350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.087806 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.087994 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.089005 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.089201 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":224,"threads_started":1}
I20260929 01:04:36.090721 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.068508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:34513)
0929 01:04:36.069338 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.069379 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.071943 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.072875 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.072884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.073687 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:36.074316 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.074335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.075547 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.075551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.076858 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.076872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.077732 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.077740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.077985 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.078810 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:36.078836 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:36.081722 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:36.085668 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:36.085679 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:36.085701 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:36.086391 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:36.086907 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:36.086914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:36.086918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:36.087211 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:36.088054 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:36.088066 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:36.088544 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.088912 (+ 368us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.089375 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":317,"threads_started":1}
I20260929 01:04:36.092712 19167 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34335
I20260929 01:04:36.093040 19167 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:36.093876 19167 heartbeater.cc:511] Master 127.3.21.254:34335 requested a full tablet report, sending...
I20260929 01:04:36.095831 18710 ts_manager.cc:194] Registered new tserver with Master: 44a8b4570e4f446691045bf93a05c015 (127.3.21.195:41277)
I20260929 01:04:36.096787 18710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:34513
I20260929 01:04:36.101464 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:36.123497 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.107066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.3.21.254:34335)
0929 01:04:36.107425 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.107430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.107892 (+ 462us) server_negotiation.cc:407] Connection header received
0929 01:04:36.108077 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.108082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.108278 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.108382 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.110133 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.111313 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.112951 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.113505 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.113626 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.116591 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.116597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.116601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.116644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.120046 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.120904 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.120909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.120911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.121054 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.121689 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.121693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.121695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.122067 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.122268 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.122868 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.123099 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260929 01:04:36.129525 18710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:04:36.138264 18710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:36.180718 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.154508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:46499 (local address 127.0.0.1:46734)
0929 01:04:36.171323 (+ 16815us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.171351 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.171791 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.172513 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.172520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.172553 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:36.173036 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.173047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.174662 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.174667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.177672 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.177688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.179749 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.179758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.179931 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.179956 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.180067 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.180208 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16672,"mutex_wait_us":119,"thread_start_us":8745,"threads_started":1}
I20260929 01:04:36.180872 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.154763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.3.21.193:43597)
0929 01:04:36.162967 (+ 8204us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.162996 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.168464 (+ 5468us) server_negotiation.cc:407] Connection header received
0929 01:04:36.170177 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.170191 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.170931 (+ 740us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.171166 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:36.172778 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.174841 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.177476 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.179110 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.179287 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.179576 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.180336 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.180529 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7955,"thread_start_us":7743,"threads_started":1}
I20260929 01:04:36.181473 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.153831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:43597 (local address 127.0.0.1:51280)
0929 01:04:36.168254 (+ 14423us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.168295 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.169885 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.171514 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.171522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.171559 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:36.171995 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.172012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.174978 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.174986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.177252 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.177267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.179359 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.179367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.179593 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.179648 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.179805 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.180044 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14200,"mutex_wait_us":30,"thread_start_us":17661,"threads_started":1}
I20260929 01:04:36.181668 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.155428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.3.21.194:46499)
0929 01:04:36.170587 (+ 15159us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.170602 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.171469 (+ 867us) server_negotiation.cc:407] Connection header received
0929 01:04:36.171993 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.171999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.172227 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.172438 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:36.173216 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.174452 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.177861 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.179429 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.179573 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.179775 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.180125 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.180377 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14808,"thread_start_us":15176,"threads_started":1}
I20260929 01:04:36.182235 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.153596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.3.21.195:41277)
0929 01:04:36.171479 (+ 17883us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.171487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.171521 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:04:36.171623 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.171631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.171915 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.172139 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:36.172919 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.174651 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.178642 (+ 3991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.179989 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.180115 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.180460 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.181688 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.181878 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17684,"thread_start_us":17493,"threads_started":1}
I20260929 01:04:36.181547 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.153260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:41277 (local address 127.0.0.1:32914)
0929 01:04:36.164725 (+ 11465us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.164804 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.165700 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.172134 (+ 6434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.172139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.172162 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:36.172625 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.172638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.174798 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.174804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.177672 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.177688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.180621 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.180629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.180818 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.180861 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.180960 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.181113 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10580,"mutex_wait_us":222,"thread_start_us":15198,"threads_started":1}
I20260929 01:04:36.186287 18968 tablet_service.cc:1467] Processing CreateTablet for tablet 390e0c4db4304c798aa5cd3dcc884a0d (DEFAULT_TABLE table=test-table [id=44ddb4fc346344de95d275b764ebfd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:36.186270 18834 tablet_service.cc:1467] Processing CreateTablet for tablet 390e0c4db4304c798aa5cd3dcc884a0d (DEFAULT_TABLE table=test-table [id=44ddb4fc346344de95d275b764ebfd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:36.187098 19102 tablet_service.cc:1467] Processing CreateTablet for tablet 390e0c4db4304c798aa5cd3dcc884a0d (DEFAULT_TABLE table=test-table [id=44ddb4fc346344de95d275b764ebfd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:36.187906 18968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e0c4db4304c798aa5cd3dcc884a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:36.187955 18834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e0c4db4304c798aa5cd3dcc884a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:36.188525 19102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390e0c4db4304c798aa5cd3dcc884a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:36.200664 19186 tablet_bootstrap.cc:492] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Bootstrap starting.
I20260929 01:04:36.204475 19187 tablet_bootstrap.cc:492] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Bootstrap starting.
I20260929 01:04:36.205508 19186 tablet_bootstrap.cc:654] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:36.206404 19188 tablet_bootstrap.cc:492] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Bootstrap starting.
I20260929 01:04:36.206993 19186 log.cc:824] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:36.208675 19187 tablet_bootstrap.cc:654] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:36.209589 19188 tablet_bootstrap.cc:654] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:36.210465 19187 log.cc:824] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:36.210708 19188 log.cc:824] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:36.215413 19188 tablet_bootstrap.cc:492] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: No bootstrap required, opened a new log
I20260929 01:04:36.215623 19186 tablet_bootstrap.cc:492] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: No bootstrap required, opened a new log
I20260929 01:04:36.215616 19187 tablet_bootstrap.cc:492] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: No bootstrap required, opened a new log
I20260929 01:04:36.215832 19188 ts_tablet_manager.cc:1397] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260929 01:04:36.215930 19186 ts_tablet_manager.cc:1397] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260929 01:04:36.215930 19187 ts_tablet_manager.cc:1397] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20260929 01:04:36.224817 19188 raft_consensus.cc:348] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.225392 19188 raft_consensus.cc:374] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:36.225553 19188 raft_consensus.cc:729] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abad13a5625b48bfbff8b5f2f96b5450, State: Initialized, Role: FOLLOWER
I20260929 01:04:36.226275 19188 consensus_queue.cc:260] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.227460 19187 raft_consensus.cc:348] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.228040 19187 raft_consensus.cc:374] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:36.228199 19187 raft_consensus.cc:729] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a8b4570e4f446691045bf93a05c015, State: Initialized, Role: FOLLOWER
I20260929 01:04:36.228394 19188 ts_tablet_manager.cc:1428] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260929 01:04:36.228407 19186 raft_consensus.cc:348] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.228938 19186 raft_consensus.cc:374] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:36.229048 19186 raft_consensus.cc:729] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0618370d00ae4c9087451377f65a7ff3, State: Initialized, Role: FOLLOWER
I20260929 01:04:36.229091 19187 consensus_queue.cc:260] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.229801 19186 consensus_queue.cc:260] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.230336 19167 heartbeater.cc:503] Master 127.3.21.254:34335 was elected leader, sending a full tablet report...
I20260929 01:04:36.231204 19033 heartbeater.cc:503] Master 127.3.21.254:34335 was elected leader, sending a full tablet report...
I20260929 01:04:36.231984 19186 ts_tablet_manager.cc:1428] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 01:04:36.233397 19187 ts_tablet_manager.cc:1428] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260929 01:04:36.266001 19034 tablet.cc:2369] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:36.316818 19168 tablet.cc:2369] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:36.404495 19192 raft_consensus.cc:482] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:36.404925 19192 raft_consensus.cc:504] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.406926 19192 leader_election.cc:288] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a8b4570e4f446691045bf93a05c015 (127.3.21.195:41277), 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499)
I20260929 01:04:36.416100 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.407548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:33821 (local address 127.3.21.195:41277)
0929 01:04:36.407784 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.407791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.408519 (+ 728us) server_negotiation.cc:407] Connection header received
0929 01:04:36.408939 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.408953 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.409032 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.409311 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:36.410003 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.411472 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.413885 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.415065 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.415257 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.415335 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.415707 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.415849 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:04:36.416114 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.407541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:54521 (local address 127.3.21.194:46499)
0929 01:04:36.407776 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.407784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.409133 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 01:04:36.409371 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.409380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.409457 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.409557 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:36.410207 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.411477 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.413836 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.414948 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.415107 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.415239 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.415630 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.415810 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:04:36.416231 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.407532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:46499 (local address 127.3.21.193:54521)
0929 01:04:36.408896 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.408914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.409170 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.409584 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.409588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.409614 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:36.410032 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.410041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.411671 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.411676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.413620 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.413641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.415226 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.415235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.415442 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.415471 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.415556 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.415720 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":198,"thread_start_us":822,"threads_started":1}
I20260929 01:04:36.416312 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.407431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:41277 (local address 127.3.21.193:33821)
0929 01:04:36.408392 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.408409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.408624 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.409338 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.409345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.409377 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:36.409803 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.409821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.411676 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.411682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.413631 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.413647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.415349 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.415355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.415498 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.415524 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.415660 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.415809 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":3,"thread_start_us":589,"threads_started":1}
I20260929 01:04:36.417462 18988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e0c4db4304c798aa5cd3dcc884a0d" candidate_uuid: "abad13a5625b48bfbff8b5f2f96b5450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0618370d00ae4c9087451377f65a7ff3" is_pre_election: true
I20260929 01:04:36.417611 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e0c4db4304c798aa5cd3dcc884a0d" candidate_uuid: "abad13a5625b48bfbff8b5f2f96b5450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a8b4570e4f446691045bf93a05c015" is_pre_election: true
I20260929 01:04:36.418077 18988 raft_consensus.cc:2454] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abad13a5625b48bfbff8b5f2f96b5450 in term 0.
I20260929 01:04:36.418139 19122 raft_consensus.cc:2454] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abad13a5625b48bfbff8b5f2f96b5450 in term 0.
I20260929 01:04:36.418916 18789 leader_election.cc:302] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0618370d00ae4c9087451377f65a7ff3, 44a8b4570e4f446691045bf93a05c015, abad13a5625b48bfbff8b5f2f96b5450; no voters:
I20260929 01:04:36.419390 19192 raft_consensus.cc:2781] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:36.419600 19192 raft_consensus.cc:482] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:36.419719 19192 raft_consensus.cc:3037] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:36.422302 19192 raft_consensus.cc:504] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.423193 19192 leader_election.cc:288] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [CANDIDATE]: Term 1 election: Requested vote from peers 44a8b4570e4f446691045bf93a05c015 (127.3.21.195:41277), 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499)
I20260929 01:04:36.423746 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e0c4db4304c798aa5cd3dcc884a0d" candidate_uuid: "abad13a5625b48bfbff8b5f2f96b5450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a8b4570e4f446691045bf93a05c015"
I20260929 01:04:36.423838 18988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390e0c4db4304c798aa5cd3dcc884a0d" candidate_uuid: "abad13a5625b48bfbff8b5f2f96b5450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0618370d00ae4c9087451377f65a7ff3"
I20260929 01:04:36.423990 19122 raft_consensus.cc:3037] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:36.424121 18988 raft_consensus.cc:3037] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:36.426955 18988 raft_consensus.cc:2454] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abad13a5625b48bfbff8b5f2f96b5450 in term 1.
I20260929 01:04:36.427028 19122 raft_consensus.cc:2454] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abad13a5625b48bfbff8b5f2f96b5450 in term 1.
I20260929 01:04:36.427551 18789 leader_election.cc:302] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0618370d00ae4c9087451377f65a7ff3, abad13a5625b48bfbff8b5f2f96b5450; no voters:
I20260929 01:04:36.427930 19192 raft_consensus.cc:2781] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:36.428381 19192 raft_consensus.cc:686] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 1 LEADER]: Becoming Leader. State: Replica: abad13a5625b48bfbff8b5f2f96b5450, State: Running, Role: LEADER
I20260929 01:04:36.428915 19192 consensus_queue.cc:237] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:36.434231 18708 catalog_manager.cc:5495] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 reported cstate change: term changed from 0 to 1, leader changed from <none> to abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "abad13a5625b48bfbff8b5f2f96b5450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } health_report { overall_health: UNKNOWN } } }
W20260929 01:04:36.448841 18900 tablet.cc:2369] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:36.485387 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.478644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.3.21.193:43597)
0929 01:04:36.478837 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.478842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.478981 (+ 139us) server_negotiation.cc:407] Connection header received
0929 01:04:36.479117 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.479121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.479219 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.479376 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:36.479840 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.480804 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.482749 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.483201 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.483321 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.484036 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:36.484680 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:36.484793 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.485072 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.485158 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:04:36.495048 19122 raft_consensus.cc:1261] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Refusing update from remote peer abad13a5625b48bfbff8b5f2f96b5450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:36.495041 18988 raft_consensus.cc:1261] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Refusing update from remote peer abad13a5625b48bfbff8b5f2f96b5450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:36.496440 19197 consensus_queue.cc:1035] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:36.497005 19192 consensus_queue.cc:1035] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:36.515322 19202 mvcc.cc:205] Tried to move back new op lower bound from 7334477318112100352 to 7334477317857341440. Current Snapshot: MvccSnapshot[applied={T|T < 7334477318112100352}]
I20260929 01:04:36.526961 18708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:36.527320 18708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:36.531737 18708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Sending DeleteTablet for 3 replicas of tablet 390e0c4db4304c798aa5cd3dcc884a0d
I20260929 01:04:36.533145 19102 tablet_service.cc:1514] Processing DeleteTablet for tablet 390e0c4db4304c798aa5cd3dcc884a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:36 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:32914
I20260929 01:04:36.533226 18834 tablet_service.cc:1514] Processing DeleteTablet for tablet 390e0c4db4304c798aa5cd3dcc884a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:36 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:51280
I20260929 01:04:36.534147 19210 tablet_replica.cc:330] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: stopping tablet replica
I20260929 01:04:36.534449 18968 tablet_service.cc:1514] Processing DeleteTablet for tablet 390e0c4db4304c798aa5cd3dcc884a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:36 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:46734
I20260929 01:04:36.535683 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 18771
I20260929 01:04:36.545303 19210 raft_consensus.cc:2229] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:36.546563 19210 raft_consensus.cc:2258] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:36.548565 19212 tablet_replica.cc:330] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: stopping tablet replica
I20260929 01:04:36.550026 19212 raft_consensus.cc:2229] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:36.550756 19212 raft_consensus.cc:2258] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:36.553356 19210 ts_tablet_manager.cc:1905] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:36.554316 19211 tablet_replica.cc:330] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: stopping tablet replica
I20260929 01:04:36.555896 19212 ts_tablet_manager.cc:1905] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:36.556315 19211 raft_consensus.cc:2229] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:04:36.557127 19211 raft_consensus.cc:2258] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:36.567196 19211 ts_tablet_manager.cc:1905] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:36.568056 19212 ts_tablet_manager.cc:1918] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:36.568250 19212 log.cc:1196] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/390e0c4db4304c798aa5cd3dcc884a0d
I20260929 01:04:36.568864 19212 ts_tablet_manager.cc:1939] T 390e0c4db4304c798aa5cd3dcc884a0d P 0618370d00ae4c9087451377f65a7ff3: Deleting consensus metadata
I20260929 01:04:36.572340 18696 catalog_manager.cc:4841] TS 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499): tablet 390e0c4db4304c798aa5cd3dcc884a0d (table test-table [id=44ddb4fc346344de95d275b764ebfd72]) successfully deleted
I20260929 01:04:36.574371 19210 ts_tablet_manager.cc:1918] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:36.574595 19210 log.cc:1196] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/390e0c4db4304c798aa5cd3dcc884a0d
I20260929 01:04:36.575037 19211 ts_tablet_manager.cc:1918] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:36.575165 19210 ts_tablet_manager.cc:1939] T 390e0c4db4304c798aa5cd3dcc884a0d P 44a8b4570e4f446691045bf93a05c015: Deleting consensus metadata
I20260929 01:04:36.575248 19211 log.cc:1196] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/390e0c4db4304c798aa5cd3dcc884a0d
I20260929 01:04:36.575793 19211 ts_tablet_manager.cc:1939] T 390e0c4db4304c798aa5cd3dcc884a0d P abad13a5625b48bfbff8b5f2f96b5450: Deleting consensus metadata
I20260929 01:04:36.577463 18695 catalog_manager.cc:4841] TS abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193:43597): tablet 390e0c4db4304c798aa5cd3dcc884a0d (table test-table [id=44ddb4fc346344de95d275b764ebfd72]) successfully deleted
I20260929 01:04:36.577878 18697 catalog_manager.cc:4841] TS 44a8b4570e4f446691045bf93a05c015 (127.3.21.195:41277): tablet 390e0c4db4304c798aa5cd3dcc884a0d (table test-table [id=44ddb4fc346344de95d275b764ebfd72]) successfully deleted
Sep 29 01:04:36 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:04:36.592706 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.536957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.3.21.193:43597)
0929 01:04:36.537220 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.537231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.564766 (+ 27535us) server_negotiation.cc:407] Connection header received
0929 01:04:36.565138 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.565145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.565312 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.565408 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.568001 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.569428 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.577245 (+ 7817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.578092 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.578220 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.584211 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.584234 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.584256 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.584315 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.588891 (+ 4576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.589891 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.589903 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.589907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.590126 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.590686 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.590692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.590696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.591241 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.591578 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.592011 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.592278 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:04:36.593699 18894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:36.699872 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 18771
I20260929 01:04:36.720816 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 18905
Sep 29 01:04:36 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:04:36.739621 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.721752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.3.21.194:46499)
0929 01:04:36.721921 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.721926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.721943 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:04:36.722023 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.722027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.722104 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.722188 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.723695 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.724654 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.726337 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.726955 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.727068 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.732365 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.732386 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.732403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.732464 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.736126 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.737015 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.737023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.737025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.737173 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.737790 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.737794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.737796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.738230 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.738586 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.739034 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.739266 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:04:36.740391 19028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:36.840456 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 18905
I20260929 01:04:36.860437 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19039
Sep 29 01:04:36 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:04:36.878950 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.861356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.3.21.195:41277)
0929 01:04:36.861541 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.861546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.861592 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:04:36.861653 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.861658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.861739 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.861825 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.863536 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.864431 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.866285 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.866815 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.866941 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.872237 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.872255 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.872273 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.872319 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.875516 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.876457 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.876467 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.876470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.876673 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.877180 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.877185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.877187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.877681 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.877939 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.878269 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.878643 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:04:36.879664 19162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:36.975618 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19039
I20260929 01:04:36.993928 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 18678
I20260929 01:04:36.995213 18740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:37.114682 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 18678
I20260929 01:04:57.138037 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34335
--webserver_interface=127.3.21.254
--webserver_port=37739
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:57.508781 19218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:57.509212 19218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:57.509397 19218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:57.517439 19218 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:04:57.517627 19218 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:04:57.517706 19218 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:04:57.517810 19218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:57.517902 19218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:57.518021 19218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:57.518113 19218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:57.518203 19218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:57.528949 19218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=37739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:57.530432 19218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:57.531980 19218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:04:57.557225 19218 server_base.cc:1034] running on GCE node
W20260929 01:04:57.557224 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:57.557255 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:57.557307 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:57.558733 19218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:57.560497 19218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:57.561769 19218 hybrid_clock.cc:648] HybridClock initialized: now 1790643897561677 us; error 60 us; skew 500 ppm
I20260929 01:04:57.562239 19218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:57 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643897, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:57.572969 19218 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:04:57.575152 19218 webserver.cc:460] Webserver started at http://127.3.21.254:37739/ using document root <none> and password file <none>
I20260929 01:04:57.575986 19218 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:57.576104 19218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:57.583016 19218 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:04:57.586448 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:57.588286 19218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:57.588501 19218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "b16e1d50066440d0b43b4bbc20adb428"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "4665013b9f6d6a448b98133fb029817f"
server_key_iv: "c538e737f011d32f51a505b174cc60c9"
server_key_version: "encryptionkey@0"
I20260929 01:04:57.589741 19218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:57.613459 19218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:57.618957 19218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:57.619293 19218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:57.640417 19218 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34335
I20260929 01:04:57.640462 19285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34335 every 8 connection(s)
I20260929 01:04:57.642170 19218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:57.647918 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19218
I20260929 01:04:57.648833 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:43597
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:34335
--webserver_port=36461
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:57.650326 19286 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:04:57.653947 19286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap starting.
I20260929 01:04:57.675061 19286 log.cc:824] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:57.688233 19286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:57.688827 19286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap complete.
I20260929 01:04:57.698891 19286 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:57.699848 19286 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16e1d50066440d0b43b4bbc20adb428, State: Initialized, Role: FOLLOWER
I20260929 01:04:57.700510 19286 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:57.700802 19286 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:57.700932 19286 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:57.701113 19286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:04:57.705477 19286 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:57.706029 19286 leader_election.cc:302] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16e1d50066440d0b43b4bbc20adb428; no voters:
I20260929 01:04:57.706717 19286 leader_election.cc:288] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:04:57.706866 19290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:04:57.708003 19290 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 LEADER]: Becoming Leader. State: Replica: b16e1d50066440d0b43b4bbc20adb428, State: Running, Role: LEADER
I20260929 01:04:57.708706 19290 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:04:57.709178 19286 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:57.711470 19291 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b16e1d50066440d0b43b4bbc20adb428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } } }
I20260929 01:04:57.711947 19291 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:57.714354 19292 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16e1d50066440d0b43b4bbc20adb428. Latest consensus state: current_term: 2 leader_uuid: "b16e1d50066440d0b43b4bbc20adb428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } } }
I20260929 01:04:57.714920 19292 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:57.716884 19296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:57.726761 19296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:57.728547 19296 catalog_manager.cc:1259] Loaded cluster ID: 0cb71c8e2b364b869d72c54483012479
I20260929 01:04:57.728870 19296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:57.733768 19296 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:57.736023 19296 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Loaded TSK: 0
I20260929 01:04:57.774739 19296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Generated new TSK 1
I20260929 01:04:57.775282 19296 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:04:58.019821 19288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:58.020237 19288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:58.020430 19288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:58.029575 19288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:58.029798 19288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:58.030031 19288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:04:58.042068 19288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:43597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=36461
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:58.043738 19288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:58.045531 19288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:58.061795 19316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:58.061897 19313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:58.061820 19314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:58.063493 19288 server_base.cc:1034] running on GCE node
I20260929 01:04:58.064222 19288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:58.065267 19288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:58.066737 19288 hybrid_clock.cc:648] HybridClock initialized: now 1790643898066680 us; error 46 us; skew 500 ppm
I20260929 01:04:58.067142 19288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:58 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:58.070863 19288 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:04:58.072695 19288 webserver.cc:460] Webserver started at http://127.3.21.193:36461/ using document root <none> and password file <none>
I20260929 01:04:58.073443 19288 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:58.073642 19288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:58.079969 19288 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:58.083109 19323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:58.084630 19288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:04:58.084807 19288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "abad13a5625b48bfbff8b5f2f96b5450"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "bfd7fdf9ba65d19ec546cf481e7cdeaf"
server_key_iv: "0de466ababf0490549d1ececd423ea21"
server_key_version: "encryptionkey@0"
I20260929 01:04:58.085850 19288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:58.101678 19288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:58.106993 19288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:58.107319 19288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:58.108520 19288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:58.110107 19288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:58.110216 19288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:58.110340 19288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:58.110414 19288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:58.150322 19288 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:43597
I20260929 01:04:58.150372 19435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:43597 every 8 connection(s)
I20260929 01:04:58.152230 19288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:58.154881 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19288
I20260929 01:04:58.155664 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:46499
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:34335
--webserver_port=38229
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:58 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:58.185811 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.156013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:36261)
0929 01:04:58.157527 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:58.157593 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:58.160551 (+ 2958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:58.162375 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:58.162389 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:58.163296 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:58.164157 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:58.164187 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.166929 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.166934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:58.168358 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:58.168385 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.170091 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.170097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:58.170307 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:58.171266 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:58.171293 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:58.173938 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:58.180072 (+ 6134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:58.180081 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:58.180097 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:58.180719 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:58.181474 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:58.181480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:58.181483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:58.181765 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:58.183235 (+ 1470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:58.183241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:58.183647 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 01:04:58.183913 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 01:04:58.184384 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":571,"threads_started":1}
I20260929 01:04:58.186272 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.155747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:36261 (local address 127.3.21.254:34335)
0929 01:04:58.156778 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:58.156799 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:58.157808 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 01:04:58.161285 (+ 3477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:58.161300 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:58.161791 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:58.162147 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:58.164418 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.166676 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.168574 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.169806 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.170028 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:58.174357 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:58.174378 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:58.174406 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:58.174517 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:58.179744 (+ 5227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:58.180966 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:58.180981 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:58.180992 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:58.181297 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:58.181951 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:58.181961 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:58.181968 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:58.183037 (+ 1069us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:58.183305 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:58.184249 (+ 944us) server_negotiation.cc:299] Negotiation successful
0929 01:04:58.184799 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":324,"threads_started":1}
I20260929 01:04:58.189749 19436 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34335
I20260929 01:04:58.190147 19436 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:58.191084 19436 heartbeater.cc:511] Master 127.3.21.254:34335 requested a full tablet report, sending...
I20260929 01:04:58.193558 19251 ts_manager.cc:194] Registered new tserver with Master: abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193:43597)
I20260929 01:04:58.195276 19251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:36261
W20260929 01:04:58.522406 19440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:58.522841 19440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:58.523008 19440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:58.530690 19440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:58.530843 19440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:58.531034 19440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:04:58.541354 19440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:46499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=38229
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:58.542796 19440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:58.544214 19440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:58.561473 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:58.561532 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:58.561735 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:58.562378 19440 server_base.cc:1034] running on GCE node
I20260929 01:04:58.563123 19440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:58.564278 19440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:58.565572 19440 hybrid_clock.cc:648] HybridClock initialized: now 1790643898565522 us; error 40 us; skew 500 ppm
I20260929 01:04:58.565953 19440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:58 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:58.570241 19440 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:04:58.572131 19440 webserver.cc:460] Webserver started at http://127.3.21.194:38229/ using document root <none> and password file <none>
I20260929 01:04:58.572865 19440 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:58.573022 19440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:58.579330 19440 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:04:58.582417 19455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:58.584152 19440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:04:58.584300 19440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0618370d00ae4c9087451377f65a7ff3"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "f53f50efa4a99cc81b621778b48d464a"
server_key_iv: "edd2a13becf6d66dc561185074299e7b"
server_key_version: "encryptionkey@0"
I20260929 01:04:58.585275 19440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:58.604513 19440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:58.609138 19440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:58.609493 19440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:58.610725 19440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:58.612303 19440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:58.612396 19440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:58.612488 19440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:58.612535 19440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:58.652086 19440 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:46499
I20260929 01:04:58.652138 19567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:46499 every 8 connection(s)
I20260929 01:04:58.654264 19440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:58.660822 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19440
I20260929 01:04:58.661723 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:41277
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:34335
--webserver_port=44817
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:58 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:58.682322 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.657694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:35715 (local address 127.3.21.254:34335)
0929 01:04:58.657881 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:58.657886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:58.659498 (+ 1612us) server_negotiation.cc:407] Connection header received
0929 01:04:58.662414 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:58.662420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:58.662614 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:58.662744 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:58.665347 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.666148 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.669223 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.669763 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.669885 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:58.674218 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:58.674227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:58.674235 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:58.674288 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:58.678008 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:58.679045 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:58.679055 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:58.679062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:58.679226 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:58.679695 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:58.679703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:58.679708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:58.680338 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:58.680511 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:58.681482 (+ 971us) server_negotiation.cc:299] Negotiation successful
0929 01:04:58.681810 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:04:58.683238 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.658293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:35715)
0929 01:04:58.659304 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:58.659355 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:58.662019 (+ 2664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:58.663114 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:58.663132 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:58.664073 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:58.665069 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:58.665111 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.666375 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.666384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:58.668995 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:58.669018 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:58.670090 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:58.670102 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:58.670574 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:58.671376 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:58.671404 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:58.673904 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:58.678209 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:58.678219 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:58.678235 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:58.678884 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:58.679346 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:58.679350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:58.679353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:58.679561 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:58.680522 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:58.680529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:58.680996 (+ 467us) client_negotiation.cc:769] Sending connection context
0929 01:04:58.681316 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 01:04:58.681835 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":360,"threads_started":1}
I20260929 01:04:58.684957 19568 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34335
I20260929 01:04:58.685285 19568 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:58.686110 19568 heartbeater.cc:511] Master 127.3.21.254:34335 requested a full tablet report, sending...
I20260929 01:04:58.687978 19251 ts_manager.cc:194] Registered new tserver with Master: 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499)
I20260929 01:04:58.689078 19251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:35715
W20260929 01:04:59.025278 19571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:59.025642 19571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:59.025807 19571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:59.033522 19571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:59.033701 19571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:59.033876 19571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:04:59.044396 19571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:41277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=44817
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:04:59.045905 19571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:59.047509 19571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:59.063283 19579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:59.063256 19576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:59.063309 19577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:59.064141 19571 server_base.cc:1034] running on GCE node
I20260929 01:04:59.064857 19571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:59.065873 19571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:59.067088 19571 hybrid_clock.cc:648] HybridClock initialized: now 1790643899067026 us; error 46 us; skew 500 ppm
I20260929 01:04:59.067428 19571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:04:59 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643899, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:59.070940 19571 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:04:59.072716 19571 webserver.cc:460] Webserver started at http://127.3.21.195:44817/ using document root <none> and password file <none>
I20260929 01:04:59.073467 19571 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:59.073590 19571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:59.079886 19571 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:59.083094 19586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:59.084774 19571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:04:59.084981 19571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "44a8b4570e4f446691045bf93a05c015"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "4b5b0c08d435be63cc51ac550d3d595f"
server_key_iv: "78a87290acbff24e4fa770edd45425d9"
server_key_version: "encryptionkey@0"
I20260929 01:04:59.085986 19571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:59.105683 19571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:59.111192 19571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:59.111500 19571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:59.112622 19571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:59.114100 19571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:59.114190 19571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:59.114336 19571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:59.114418 19571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:59.154441 19571 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:41277
I20260929 01:04:59.154520 19698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:41277 every 8 connection(s)
I20260929 01:04:59.156030 19571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:59.163784 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19571
Sep 29 01:04:59 dist-test-slave-k0b7 krb5kdc[18665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643899, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:04:59.181576 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.159028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:36165 (local address 127.3.21.254:34335)
0929 01:04:59.159228 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.159233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.160577 (+ 1344us) server_negotiation.cc:407] Connection header received
0929 01:04:59.162584 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.162590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.162694 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.162808 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:59.164961 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.166034 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.168167 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.168691 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.168825 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.173634 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:59.173644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:59.173650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:59.173700 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:59.177375 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.178722 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.178727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.178729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.178883 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.179542 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.179546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.179548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.179889 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:59.180000 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.180902 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.181149 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:04:59.183027 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.159437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:36165)
0929 01:04:59.160206 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.160251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:59.162220 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:59.163077 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:59.163086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:59.163888 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:59.164768 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.164791 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.166267 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.166276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.167951 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.167966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.168940 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.168947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.169309 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.170213 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:59.170241 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:59.173358 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:59.177674 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:59.177691 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:59.177715 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:59.178466 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:59.179022 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:59.179028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:59.179033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:59.179348 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:59.180050 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:59.180061 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:59.180476 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 01:04:59.180811 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 01:04:59.181380 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":292,"threads_started":1}
I20260929 01:04:59.183341 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.164929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.3.21.254:34335)
0929 01:04:59.165855 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.165859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.165875 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:04:59.165951 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.165955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.166019 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.166116 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:59.168129 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.169052 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.171044 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.171843 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.172025 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.176209 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:59.176215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:59.176219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:59.176274 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:59.179937 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.180772 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.180780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.180784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.180977 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.181489 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.181495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.181498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.181908 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:59.182218 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.182657 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.182895 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":680,"threads_started":1}
I20260929 01:04:59.184842 19699 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34335
I20260929 01:04:59.185221 19699 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:59.186190 19699 heartbeater.cc:511] Master 127.3.21.254:34335 requested a full tablet report, sending...
I20260929 01:04:59.187832 19250 ts_manager.cc:194] Registered new tserver with Master: 44a8b4570e4f446691045bf93a05c015 (127.3.21.195:41277)
I20260929 01:04:59.188850 19250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:36165
I20260929 01:04:59.198963 19436 heartbeater.cc:503] Master 127.3.21.254:34335 was elected leader, sending a full tablet report...
I20260929 01:04:59.200053 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:59.208475 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.201227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.3.21.254:34335)
0929 01:04:59.201483 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.201491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.202101 (+ 610us) server_negotiation.cc:407] Connection header received
0929 01:04:59.202330 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.202336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.202573 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.202717 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:59.203226 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.204082 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.205913 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.206388 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.206602 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.207451 (+ 849us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:59.207776 (+ 325us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 01:04:59.208163 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58644: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 01:04:59.208650 19703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58644: token expired
W20260929 01:04:59.209348 19171 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.21.254:34335)
I20260929 01:04:59.225885 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.209936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.3.21.254:34335)
0929 01:04:59.210191 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.210196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.210629 (+ 433us) server_negotiation.cc:407] Connection header received
0929 01:04:59.210694 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.210697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.210767 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.210860 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:59.212351 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.213215 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.214836 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.215310 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.215424 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.219502 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:59.219509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:59.219513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:59.219561 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:59.222472 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.223293 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.223298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.223300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.223506 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.223932 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.223936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.223938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.224338 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:59.224590 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.225161 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.225521 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:04:59.230436 19250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:04:59.239207 19250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:59.266119 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.255195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.3.21.194:46499)
0929 01:04:59.256171 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.256183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.257600 (+ 1417us) server_negotiation.cc:407] Connection header received
0929 01:04:59.257719 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.257730 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.257946 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.258145 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:59.259039 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.260709 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.263315 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.264658 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.264839 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.265087 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.265523 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.265703 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":637,"threads_started":1}
I20260929 01:04:59.266213 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.253280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:41277 (local address 127.0.0.1:57462)
0929 01:04:59.255440 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.255465 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:59.255599 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:59.257463 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:59.257472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:59.257503 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:59.258033 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.258053 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.260129 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.260137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.262991 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.263088 (+ 97us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.265178 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.265185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.265390 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.265435 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:04:59.265515 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:04:59.265667 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2009,"mutex_wait_us":58,"thread_start_us":3670,"threads_started":1}
I20260929 01:04:59.266701 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.253268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.3.21.193:43597)
0929 01:04:59.257203 (+ 3935us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.257212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.257244 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:04:59.257355 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.257362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.257619 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.257826 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:59.258538 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.260398 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.263390 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.264928 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.265078 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.265319 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.266037 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.266281 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3357,"thread_start_us":3295,"threads_started":1}
I20260929 01:04:59.266966 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.254550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:46499 (local address 127.0.0.1:47046)
0929 01:04:59.257258 (+ 2708us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.257280 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:59.257430 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:59.258162 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:59.258168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:59.258194 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:59.258791 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.258805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.260854 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.260863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.263098 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.263113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.264917 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.264925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.265235 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.265277 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 01:04:59.265438 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 01:04:59.265670 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2559,"mutex_wait_us":88,"thread_start_us":807,"threads_started":1}
I20260929 01:04:59.266346 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.253077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:43597 (local address 127.0.0.1:59868)
0929 01:04:59.254718 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.254783 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:59.255007 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:59.257821 (+ 2814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:59.257825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:59.257851 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:59.258321 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.258333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.260573 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.260579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.263187 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.263205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.265462 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.265467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.265627 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.265672 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:04:59.265750 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 01:04:59.265893 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1250,"mutex_wait_us":137,"thread_start_us":1040,"threads_started":1}
I20260929 01:04:59.268689 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.253898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.3.21.195:41277)
0929 01:04:59.256749 (+ 2851us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.256761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.256805 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:04:59.256933 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.256945 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.257226 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.257459 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:59.258317 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.259976 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.263335 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.264928 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.266656 (+ 1728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.267068 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.267169 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.267363 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2640,"thread_start_us":2425,"threads_started":1}
I20260929 01:04:59.271169 19503 tablet_service.cc:1467] Processing CreateTablet for tablet 86385522b6254cfdacf2a039c0502dc2 (DEFAULT_TABLE table=test-table [id=d4db4b07b0b848bb801b00ac8fd54a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:59.271279 19371 tablet_service.cc:1467] Processing CreateTablet for tablet 86385522b6254cfdacf2a039c0502dc2 (DEFAULT_TABLE table=test-table [id=d4db4b07b0b848bb801b00ac8fd54a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:59.272550 19503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86385522b6254cfdacf2a039c0502dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:59.272763 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86385522b6254cfdacf2a039c0502dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:59.272284 19634 tablet_service.cc:1467] Processing CreateTablet for tablet 86385522b6254cfdacf2a039c0502dc2 (DEFAULT_TABLE table=test-table [id=d4db4b07b0b848bb801b00ac8fd54a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:59.273703 19634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86385522b6254cfdacf2a039c0502dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:59.286809 19715 tablet_bootstrap.cc:492] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Bootstrap starting.
I20260929 01:04:59.290553 19716 tablet_bootstrap.cc:492] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Bootstrap starting.
I20260929 01:04:59.293143 19715 tablet_bootstrap.cc:654] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:59.294672 19716 tablet_bootstrap.cc:654] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:59.294761 19715 log.cc:824] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:59.294955 19717 tablet_bootstrap.cc:492] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Bootstrap starting.
I20260929 01:04:59.296057 19716 log.cc:824] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:59.299100 19717 tablet_bootstrap.cc:654] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:59.299526 19716 tablet_bootstrap.cc:492] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: No bootstrap required, opened a new log
I20260929 01:04:59.299537 19715 tablet_bootstrap.cc:492] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: No bootstrap required, opened a new log
I20260929 01:04:59.299881 19715 ts_tablet_manager.cc:1397] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 01:04:59.299901 19716 ts_tablet_manager.cc:1397] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260929 01:04:59.300544 19717 log.cc:824] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:59.304374 19717 tablet_bootstrap.cc:492] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: No bootstrap required, opened a new log
I20260929 01:04:59.304701 19717 ts_tablet_manager.cc:1397] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 01:04:59.309728 19715 raft_consensus.cc:348] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.310331 19715 raft_consensus.cc:374] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:59.310451 19715 raft_consensus.cc:729] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0618370d00ae4c9087451377f65a7ff3, State: Initialized, Role: FOLLOWER
I20260929 01:04:59.310642 19716 raft_consensus.cc:348] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.311098 19716 raft_consensus.cc:374] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:59.311188 19716 raft_consensus.cc:729] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abad13a5625b48bfbff8b5f2f96b5450, State: Initialized, Role: FOLLOWER
I20260929 01:04:59.311367 19715 consensus_queue.cc:260] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.311756 19716 consensus_queue.cc:260] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.312778 19568 heartbeater.cc:503] Master 127.3.21.254:34335 was elected leader, sending a full tablet report...
I20260929 01:04:59.314934 19715 ts_tablet_manager.cc:1428] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260929 01:04:59.316747 19716 ts_tablet_manager.cc:1428] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260929 01:04:59.317142 19717 raft_consensus.cc:348] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.317644 19717 raft_consensus.cc:374] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:59.317791 19717 raft_consensus.cc:729] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a8b4570e4f446691045bf93a05c015, State: Initialized, Role: FOLLOWER
I20260929 01:04:59.318629 19717 consensus_queue.cc:260] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.319918 19699 heartbeater.cc:503] Master 127.3.21.254:34335 was elected leader, sending a full tablet report...
I20260929 01:04:59.320550 19717 ts_tablet_manager.cc:1428] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260929 01:04:59.377038 19723 raft_consensus.cc:482] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:59.377418 19723 raft_consensus.cc:504] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.379515 19723 leader_election.cc:288] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193:43597), 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499)
I20260929 01:04:59.387781 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.379850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:46499 (local address 127.3.21.195:54033)
0929 01:04:59.380448 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.380481 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:59.380696 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:59.381182 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:59.381188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:59.381241 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:59.381446 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.381459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.383036 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.383043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.385086 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.385102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.386799 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.386807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.387088 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.387131 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:04:59.387261 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 01:04:59.387430 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":45}
I20260929 01:04:59.387786 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.380170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:54033 (local address 127.3.21.194:46499)
0929 01:04:59.380375 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.380382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.380614 (+ 232us) server_negotiation.cc:407] Connection header received
0929 01:04:59.380881 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.380887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.380974 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.381098 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:59.381682 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.382826 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.385273 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.386635 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.386767 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.386826 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.387334 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.387474 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:04:59.389453 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.380099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:43597 (local address 127.3.21.195:37675)
0929 01:04:59.381011 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.381030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:59.381166 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:59.381770 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:59.381778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:59.381807 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:59.382185 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.382197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.383737 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.383743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.386169 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:59.386186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.388390 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.388397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:59.388602 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.388632 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:04:59.388733 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:04:59.388844 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":1,"thread_start_us":548,"threads_started":1}
I20260929 01:04:59.389217 19523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86385522b6254cfdacf2a039c0502dc2" candidate_uuid: "44a8b4570e4f446691045bf93a05c015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0618370d00ae4c9087451377f65a7ff3" is_pre_election: true
I20260929 01:04:59.389462 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.380115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:37675 (local address 127.3.21.193:43597)
0929 01:04:59.380339 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.380346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.381144 (+ 798us) server_negotiation.cc:407] Connection header received
0929 01:04:59.381323 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.381329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.381566 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.381666 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:59.382370 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.383549 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.386396 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.387933 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.388153 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.388244 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.388796 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.388988 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:04:59.389919 19523 raft_consensus.cc:2454] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a8b4570e4f446691045bf93a05c015 in term 0.
I20260929 01:04:59.390759 19589 leader_election.cc:302] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0618370d00ae4c9087451377f65a7ff3, 44a8b4570e4f446691045bf93a05c015; no voters:
I20260929 01:04:59.391199 19723 raft_consensus.cc:2781] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:59.391328 19723 raft_consensus.cc:482] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:59.391489 19723 raft_consensus.cc:3037] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:59.391323 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86385522b6254cfdacf2a039c0502dc2" candidate_uuid: "44a8b4570e4f446691045bf93a05c015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abad13a5625b48bfbff8b5f2f96b5450" is_pre_election: true
I20260929 01:04:59.392028 19391 raft_consensus.cc:2454] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a8b4570e4f446691045bf93a05c015 in term 0.
I20260929 01:04:59.394037 19723 raft_consensus.cc:504] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
I20260929 01:04:59.394843 19723 leader_election.cc:288] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [CANDIDATE]: Term 1 election: Requested vote from peers abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193:43597), 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499)
I20260929 01:04:59.395355 19391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86385522b6254cfdacf2a039c0502dc2" candidate_uuid: "44a8b4570e4f446691045bf93a05c015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abad13a5625b48bfbff8b5f2f96b5450"
I20260929 01:04:59.395602 19391 raft_consensus.cc:3037] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:59.395632 19523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86385522b6254cfdacf2a039c0502dc2" candidate_uuid: "44a8b4570e4f446691045bf93a05c015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0618370d00ae4c9087451377f65a7ff3"
I20260929 01:04:59.395859 19523 raft_consensus.cc:3037] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:59.399127 19391 raft_consensus.cc:2454] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a8b4570e4f446691045bf93a05c015 in term 1.
I20260929 01:04:59.399196 19523 raft_consensus.cc:2454] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a8b4570e4f446691045bf93a05c015 in term 1.
I20260929 01:04:59.399698 19588 leader_election.cc:302] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a8b4570e4f446691045bf93a05c015, abad13a5625b48bfbff8b5f2f96b5450; no voters:
I20260929 01:04:59.400120 19723 raft_consensus.cc:2781] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:59.400663 19723 raft_consensus.cc:686] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 1 LEADER]: Becoming Leader. State: Replica: 44a8b4570e4f446691045bf93a05c015, State: Running, Role: LEADER
I20260929 01:04:59.401196 19723 consensus_queue.cc:237] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } }
W20260929 01:04:59.404911 19437 tablet.cc:2369] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:59.407244 19569 tablet.cc:2369] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:59.406888 19251 catalog_manager.cc:5495] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44a8b4570e4f446691045bf93a05c015 (127.3.21.195). New cstate: current_term: 1 leader_uuid: "44a8b4570e4f446691045bf93a05c015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a8b4570e4f446691045bf93a05c015" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 41277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 } health_report { overall_health: UNKNOWN } } }
W20260929 01:04:59.407856 19700 tablet.cc:2369] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:59.429080 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.422735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.3.21.195:41277)
0929 01:04:59.422926 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.422930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.422949 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:04:59.423089 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.423093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.423172 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.423306 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:59.423665 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.424498 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.426122 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.426776 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.426910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.427535 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:59.428141 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:59.428293 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.428763 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.428854 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:04:59.440145 19523 raft_consensus.cc:1261] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Refusing update from remote peer 44a8b4570e4f446691045bf93a05c015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:59.440150 19391 raft_consensus.cc:1261] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Refusing update from remote peer 44a8b4570e4f446691045bf93a05c015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:59.441843 19723 consensus_queue.cc:1035] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0618370d00ae4c9087451377f65a7ff3" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 46499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:59.442296 19725 consensus_queue.cc:1035] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abad13a5625b48bfbff8b5f2f96b5450" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:59.458475 19728 mvcc.cc:205] Tried to move back new op lower bound from 7334477412093198336 to 7334477411951497216. Current Snapshot: MvccSnapshot[applied={T|T < 7334477412093198336}]
I20260929 01:04:59.474709 19250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:59.475075 19250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:59.479645 19250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Sending DeleteTablet for 3 replicas of tablet 86385522b6254cfdacf2a039c0502dc2
I20260929 01:04:59.481082 19503 tablet_service.cc:1514] Processing DeleteTablet for tablet 86385522b6254cfdacf2a039c0502dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:59 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:47046
I20260929 01:04:59.481082 19371 tablet_service.cc:1514] Processing DeleteTablet for tablet 86385522b6254cfdacf2a039c0502dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:59 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:59868
I20260929 01:04:59.481106 19634 tablet_service.cc:1514] Processing DeleteTablet for tablet 86385522b6254cfdacf2a039c0502dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:59 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:57462
I20260929 01:04:59.483810 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19288
I20260929 01:04:59.494508 19740 tablet_replica.cc:330] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: stopping tablet replica
I20260929 01:04:59.495147 19740 raft_consensus.cc:2229] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:59.495844 19740 raft_consensus.cc:2258] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:59.496867 19738 tablet_replica.cc:330] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: stopping tablet replica
I20260929 01:04:59.497403 19738 raft_consensus.cc:2229] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:04:59.498755 19739 tablet_replica.cc:330] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: stopping tablet replica
I20260929 01:04:59.498818 19738 raft_consensus.cc:2258] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:59.499269 19739 raft_consensus.cc:2229] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:59.499867 19739 raft_consensus.cc:2258] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:59.500206 19740 ts_tablet_manager.cc:1905] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:59.501397 19738 ts_tablet_manager.cc:1905] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:59.502107 19739 ts_tablet_manager.cc:1905] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:59.508893 19740 ts_tablet_manager.cc:1918] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:59.509130 19740 log.cc:1196] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/86385522b6254cfdacf2a039c0502dc2
I20260929 01:04:59.509799 19740 ts_tablet_manager.cc:1939] T 86385522b6254cfdacf2a039c0502dc2 P 0618370d00ae4c9087451377f65a7ff3: Deleting consensus metadata
I20260929 01:04:59.510402 19738 ts_tablet_manager.cc:1918] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:59.510459 19739 ts_tablet_manager.cc:1918] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:59.510684 19739 log.cc:1196] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/86385522b6254cfdacf2a039c0502dc2
I20260929 01:04:59.510684 19738 log.cc:1196] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/86385522b6254cfdacf2a039c0502dc2
I20260929 01:04:59.511348 19739 ts_tablet_manager.cc:1939] T 86385522b6254cfdacf2a039c0502dc2 P abad13a5625b48bfbff8b5f2f96b5450: Deleting consensus metadata
I20260929 01:04:59.511380 19738 ts_tablet_manager.cc:1939] T 86385522b6254cfdacf2a039c0502dc2 P 44a8b4570e4f446691045bf93a05c015: Deleting consensus metadata
I20260929 01:04:59.512639 19237 catalog_manager.cc:4841] TS 0618370d00ae4c9087451377f65a7ff3 (127.3.21.194:46499): tablet 86385522b6254cfdacf2a039c0502dc2 (table test-table [id=d4db4b07b0b848bb801b00ac8fd54a01]) successfully deleted
I20260929 01:04:59.513263 19236 catalog_manager.cc:4841] TS abad13a5625b48bfbff8b5f2f96b5450 (127.3.21.193:43597): tablet 86385522b6254cfdacf2a039c0502dc2 (table test-table [id=d4db4b07b0b848bb801b00ac8fd54a01]) successfully deleted
I20260929 01:04:59.513396 19238 catalog_manager.cc:4841] TS 44a8b4570e4f446691045bf93a05c015 (127.3.21.195:41277): tablet 86385522b6254cfdacf2a039c0502dc2 (table test-table [id=d4db4b07b0b848bb801b00ac8fd54a01]) successfully deleted
I20260929 01:04:59.522044 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.485918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.3.21.193:43597)
0929 01:04:59.486210 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.486220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.486246 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:04:59.486333 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.486342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.486504 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.486649 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:59.490206 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.492154 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.495185 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.496214 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.496389 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.513295 (+ 16906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:59.513311 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:59.513330 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:59.513380 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:59.517930 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.519046 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.519059 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.519063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.519269 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.519877 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.519884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.519888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.520450 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:59.520829 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.521277 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.521588 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 01:04:59.523082 19431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:59.633356 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19288
I20260929 01:04:59.679661 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19440
I20260929 01:04:59.697896 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.681487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.3.21.194:46499)
0929 01:04:59.681718 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.681726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.681750 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:04:59.681821 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.681828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.681964 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.682080 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:59.684467 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.685291 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.686675 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.687114 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.687260 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.690635 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:59.690658 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:59.690681 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:59.690736 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:59.694126 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.695161 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.695170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.695172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.695319 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.695994 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.696002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.696004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.696516 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:59.696950 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.697348 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.697575 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 01:04:59.698765 19563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:59.802439 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19440
I20260929 01:04:59.829329 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19571
I20260929 01:04:59.850404 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.830214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.3.21.195:41277)
0929 01:04:59.830438 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:59.830445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:59.830525 (+ 80us) server_negotiation.cc:407] Connection header received
0929 01:04:59.830782 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:59.830789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:59.830888 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:59.831042 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:59.833435 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.834909 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.836541 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:59.837299 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:59.837493 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:59.840671 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:59.840700 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:59.840730 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:59.840819 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:59.846344 (+ 5525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.847799 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.847815 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.847818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.848190 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:59.848638 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:59.848644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:59.848647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:59.849054 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:59.849401 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:59.849741 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 01:04:59.849995 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:04:59.851351 19694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:04:59.986083 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19571
I20260929 01:05:00.009912 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19218
I20260929 01:05:00.011756 19281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:00.131873 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19218
I20260929 01:05:20.155849 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34335
--webserver_interface=127.3.21.254
--webserver_port=37739
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:20.516415 19745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:20.516955 19745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:20.517261 19745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:20.525120 19745 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:05:20.525245 19745 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:05:20.525316 19745 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:05:20.525393 19745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:20.525486 19745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:20.525586 19745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:20.525666 19745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:20.525719 19745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:20.536396 19745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=37739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:20.537804 19745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:20.539247 19745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:20.552575 19750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:20.552623 19753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:20.552680 19751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:20.560911 19745 server_base.cc:1034] running on GCE node
I20260929 01:05:20.562032 19745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:20.563931 19745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:20.565238 19745 hybrid_clock.cc:648] HybridClock initialized: now 1790643920565173 us; error 44 us; skew 500 ppm
I20260929 01:05:20.565657 19745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:20 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643920, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:20.569597 19745 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:05:20.571491 19745 webserver.cc:460] Webserver started at http://127.3.21.254:37739/ using document root <none> and password file <none>
I20260929 01:05:20.572396 19745 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:20.572551 19745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:20.578679 19745 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:20.582028 19760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:20.583623 19745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:05:20.583868 19745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "b16e1d50066440d0b43b4bbc20adb428"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "4665013b9f6d6a448b98133fb029817f"
server_key_iv: "c538e737f011d32f51a505b174cc60c9"
server_key_version: "encryptionkey@0"
I20260929 01:05:20.584882 19745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:20.620350 19745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:20.625999 19745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:20.626317 19745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:20.647207 19745 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34335
I20260929 01:05:20.647264 19811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34335 every 8 connection(s)
I20260929 01:05:20.648861 19745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:20.656157 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19745
I20260929 01:05:20.656349 19812 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:05:20.657074 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:43597
--local_ip_for_outbound_sockets=127.3.21.193
--tserver_master_addrs=127.3.21.254:34335
--webserver_port=36461
--webserver_interface=127.3.21.193
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:20.659953 19812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap starting.
I20260929 01:05:20.690912 19812 log.cc:824] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:20.712306 19812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:05:20.712968 19812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Bootstrap complete.
I20260929 01:05:20.723858 19812 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:05:20.724839 19812 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16e1d50066440d0b43b4bbc20adb428, State: Initialized, Role: FOLLOWER
I20260929 01:05:20.725561 19812 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:05:20.725865 19812 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:20.726011 19812 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:20.726235 19812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:05:20.730762 19812 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:05:20.731381 19812 leader_election.cc:302] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16e1d50066440d0b43b4bbc20adb428; no voters:
I20260929 01:05:20.732146 19812 leader_election.cc:288] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:05:20.732307 19816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:05:20.733494 19816 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [term 3 LEADER]: Becoming Leader. State: Replica: b16e1d50066440d0b43b4bbc20adb428, State: Running, Role: LEADER
I20260929 01:05:20.734298 19816 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } }
I20260929 01:05:20.734694 19812 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:20.737332 19817 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b16e1d50066440d0b43b4bbc20adb428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } } }
I20260929 01:05:20.737320 19818 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16e1d50066440d0b43b4bbc20adb428. Latest consensus state: current_term: 3 leader_uuid: "b16e1d50066440d0b43b4bbc20adb428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16e1d50066440d0b43b4bbc20adb428" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34335 } } }
I20260929 01:05:20.737830 19818 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:20.737835 19817 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:20.751134 19822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:20.761430 19822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:20.764267 19822 catalog_manager.cc:1259] Loaded cluster ID: 0cb71c8e2b364b869d72c54483012479
I20260929 01:05:20.764487 19822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:20.771078 19822 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:20.773425 19822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Loaded TSK: 0
I20260929 01:05:20.773667 19822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b16e1d50066440d0b43b4bbc20adb428: Loaded TSK: 1
F20260929 01:05:20.775074 19822 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0ef4ba09ad google::LogMessage::Fail() at ??:0
@ 0x7f0ef4ba388a google::LogMessage::SendToLog() at ??:0
@ 0x7f0ef4ba04de google::LogMessage::Flush() at ??:0
@ 0x7f0ef4ba1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0ef89047d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0ef88f373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0f0a5cf1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0f0a5ce783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0f0a65a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f0f0a65a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0f0a6a45e9 std::function<>::operator()() at ??:0
@ 0x7f0f0a5d2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0f0a5d0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0f0a658610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0f0a658439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0ef77ec345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0ef77ef91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0ef77ef799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0ef77caeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0efaaa46db start_thread at ??:0
@ 0x7f0ef3a3a61f clone at ??:0
W20260929 01:05:21.156355 19814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:21.156736 19814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:21.156904 19814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:21.165179 19814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:21.165366 19814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:21.165570 19814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:21.176306 19814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:43597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=36461
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:21.177781 19814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:21.179180 19814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:21.194063 19860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:21.194128 19859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:21.194128 19862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:21.195873 19814 server_base.cc:1034] running on GCE node
I20260929 01:05:21.196743 19814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:21.197942 19814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:21.199232 19814 hybrid_clock.cc:648] HybridClock initialized: now 1790643921199160 us; error 60 us; skew 500 ppm
I20260929 01:05:21.199647 19814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:21 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643921, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:21.203845 19814 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:05:21.205833 19814 webserver.cc:460] Webserver started at http://127.3.21.193:36461/ using document root <none> and password file <none>
I20260929 01:05:21.206698 19814 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:21.206820 19814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:21.213140 19814 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:05:21.216574 19869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:21.218371 19814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:05:21.218634 19814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "abad13a5625b48bfbff8b5f2f96b5450"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-k0b7"
server_key: "bfd7fdf9ba65d19ec546cf481e7cdeaf"
server_key_iv: "0de466ababf0490549d1ececd423ea21"
server_key_version: "encryptionkey@0"
I20260929 01:05:21.219779 19814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:21.241534 19814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:21.247094 19814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:21.247452 19814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:21.248521 19814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:21.250442 19814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:21.250568 19814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:05:21.250654 19814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:21.250712 19814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:21.294507 19814 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:43597
I20260929 01:05:21.294559 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:43597 every 8 connection(s)
I20260929 01:05:21.296111 19814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:21.301487 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:21.299264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:35015)
0929 01:05:21.300263 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:21.300488 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":460,"threads_started":1}
W20260929 01:05:21.302268 19982 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:34335 (0 consecutive failures): Network error: Failed to ping master at 127.3.21.254:34335: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
I20260929 01:05:21.303349 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:21.302890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:43277)
0929 01:05:21.303065 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:21.303178 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 01:05:21.304319 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:21.303917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:33187)
0929 01:05:21.304073 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:21.304159 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 01:05:21.304754 19982 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:05:21.306735 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19814
I20260929 01:05:21.307484 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:46499
--local_ip_for_outbound_sockets=127.3.21.194
--tserver_master_addrs=127.3.21.254:34335
--webserver_port=38229
--webserver_interface=127.3.21.194
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:21.678574 19985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:21.678893 19985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:21.679080 19985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:21.686955 19985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:21.687080 19985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:21.687283 19985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:05:21.698292 19985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:46499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=38229
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:21.699831 19985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:21.701246 19985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:21.718063 19990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:21.718266 19985 server_base.cc:1034] running on GCE node
W20260929 01:05:21.718087 19991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:21.718110 19993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:21.719638 19985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:21.720741 19985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:21.722029 19985 hybrid_clock.cc:648] HybridClock initialized: now 1790643921721940 us; error 80 us; skew 500 ppm
I20260929 01:05:21.722456 19985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:21 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643921, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:21.726643 19985 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:05:21.728577 19985 webserver.cc:460] Webserver started at http://127.3.21.194:38229/ using document root <none> and password file <none>
I20260929 01:05:21.729434 19985 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:21.729589 19985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:21.736248 19985 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:05:21.739835 20000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:21.741632 19985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:05:21.741865 19985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "0618370d00ae4c9087451377f65a7ff3"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "f53f50efa4a99cc81b621778b48d464a"
server_key_iv: "edd2a13becf6d66dc561185074299e7b"
server_key_version: "encryptionkey@0"
I20260929 01:05:21.743109 19985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:21.763280 19985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:21.768749 19985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:21.769090 19985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:21.770421 19985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:21.772158 19985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:21.772300 19985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:21.772449 19985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:21.772549 19985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:21.817163 19985 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:46499
I20260929 01:05:21.817282 20112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:46499 every 8 connection(s)
I20260929 01:05:21.818923 19985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:21.824796 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:21.822286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:54983)
0929 01:05:21.823303 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:21.823738 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":442,"threads_started":1}
W20260929 01:05:21.825986 20113 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:34335 (0 consecutive failures): Network error: Failed to ping master at 127.3.21.254:34335: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
I20260929 01:05:21.827379 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 19985
I20260929 01:05:21.827904 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:21.827349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:36185)
0929 01:05:21.827637 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:21.827757 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 01:05:21.828047 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:41277
--local_ip_for_outbound_sockets=127.3.21.195
--tserver_master_addrs=127.3.21.254:34335
--webserver_port=44817
--webserver_interface=127.3.21.195
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:21.829265 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:21.828842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:44995)
0929 01:05:21.829016 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:21.829122 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 01:05:21.829675 20113 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:05:22.198165 20116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:22.198597 20116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:22.198797 20116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:22.206786 20116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:22.206974 20116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:22.207165 20116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:05:22.217875 20116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:41277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=44817
--tserver_master_addrs=127.3.21.254:34335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:22.219558 20116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:22.221092 20116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:22.237964 20122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:22.237938 20124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:22.237979 20121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:22.240269 20116 server_base.cc:1034] running on GCE node
I20260929 01:05:22.241035 20116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:22.242173 20116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:22.243448 20116 hybrid_clock.cc:648] HybridClock initialized: now 1790643922243374 us; error 72 us; skew 500 ppm
I20260929 01:05:22.243836 20116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:22 dist-test-slave-k0b7 krb5kdc[18665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643922, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:22.247988 20116 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:05:22.249747 20116 webserver.cc:460] Webserver started at http://127.3.21.195:44817/ using document root <none> and password file <none>
I20260929 01:05:22.250586 20116 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:22.250748 20116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:22.257841 20116 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:05:22.261596 20131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:22.263564 20116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:22.263833 20116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "44a8b4570e4f446691045bf93a05c015"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-k0b7"
server_key: "4b5b0c08d435be63cc51ac550d3d595f"
server_key_iv: "78a87290acbff24e4fa770edd45425d9"
server_key_version: "encryptionkey@0"
I20260929 01:05:22.265260 20116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:22.297715 20116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:22.303853 20116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:22.304355 20116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:22.305903 20116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:22.306686 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:22.305415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:32919)
0929 01:05:22.306152 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:22.306301 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":404,"threads_started":1}
I20260929 01:05:22.307894 20116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:22.307994 20116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:22.308077 20116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:22.308130 20116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:22.351389 20116 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:41277
I20260929 01:05:22.351572 20244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:41277 every 8 connection(s)
I20260929 01:05:22.353049 20116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:22.358460 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:22.356518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:37531)
0929 01:05:22.357261 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:22.357495 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
I20260929 01:05:22.358639 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20116
W20260929 01:05:22.359319 20245 heartbeater.cc:644] Failed to heartbeat to 127.3.21.254:34335 (0 consecutive failures): Network error: Failed to ping master at 127.3.21.254:34335: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
I20260929 01:05:22.360178 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:22.359837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:41153)
0929 01:05:22.359990 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:22.360071 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:05:22.361148 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:22.360817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:54791)
0929 01:05:22.360961 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:22.361044 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:05:22.361423 20245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:05:22.831871 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:22.830794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:60407)
0929 01:05:22.831411 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:22.831556 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":344,"threads_started":1}
I20260929 01:05:23.308924 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:23.307987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:41497)
0929 01:05:23.308631 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:23.308730 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":373,"threads_started":1}
I20260929 01:05:23.363306 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:23.362240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:48971)
0929 01:05:23.362890 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:23.362991 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":377,"threads_started":1}
I20260929 01:05:23.834224 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:23.833089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:33569)
0929 01:05:23.833740 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:23.833954 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260929 01:05:24.311204 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:24.310250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:53693)
0929 01:05:24.310882 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:24.311010 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":377,"threads_started":1}
I20260929 01:05:24.365221 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:24.364352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:40251)
0929 01:05:24.364931 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:24.365020 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":252,"threads_started":1}
I20260929 01:05:24.836257 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:24.835492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:40981)
0929 01:05:24.835982 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:24.836069 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260929 01:05:25.313395 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:25.312446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:37737)
0929 01:05:25.313010 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:25.313103 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":316,"threads_started":1}
I20260929 01:05:25.367376 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:25.366352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:52001)
0929 01:05:25.367088 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:25.367180 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":497,"threads_started":1}
I20260929 01:05:25.838238 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:25.837323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:34351)
0929 01:05:25.837953 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:25.838046 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":308,"threads_started":1}
I20260929 01:05:26.315685 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.314702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:53291)
0929 01:05:26.315240 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.315400 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260929 01:05:26.369446 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.368483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:54529)
0929 01:05:26.369092 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.369186 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260929 01:05:26.840276 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.839403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:33703)
0929 01:05:26.839997 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.840087 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":245,"threads_started":1}
I20260929 01:05:27.318186 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.317108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:39769)
0929 01:05:27.317618 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.317843 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
I20260929 01:05:27.371433 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.370619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:46845)
0929 01:05:27.371126 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.371207 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
I20260929 01:05:27.842343 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.841453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:44885)
0929 01:05:27.842073 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.842171 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":359,"threads_started":1}
I20260929 01:05:28.320523 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:28.319513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:50285)
0929 01:05:28.320209 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:28.320295 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":411,"threads_started":1}
I20260929 01:05:28.373479 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:28.372620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:45961)
0929 01:05:28.373190 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:28.373304 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20260929 01:05:28.844863 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:28.843577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:37603)
0929 01:05:28.844404 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:28.844508 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":387,"threads_started":1}
I20260929 01:05:29.322757 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:29.321660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:38913)
0929 01:05:29.322351 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:29.322449 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":415,"threads_started":1}
I20260929 01:05:29.375710 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:29.374672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:55811)
0929 01:05:29.375399 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:29.375494 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":260,"threads_started":1}
I20260929 01:05:29.847394 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:29.846198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:42957)
0929 01:05:29.846977 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:29.847126 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":346,"threads_started":1}
I20260929 01:05:30.324987 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:30.324066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:50093)
0929 01:05:30.324623 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:30.324722 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":255,"threads_started":1}
I20260929 01:05:30.378001 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:30.377113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:33619)
0929 01:05:30.377660 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:30.377791 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":281,"threads_started":1}
I20260929 01:05:30.849779 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:30.848590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:46471)
0929 01:05:30.849491 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:30.849582 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":610,"threads_started":1}
I20260929 01:05:31.327101 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:31.326183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:52217)
0929 01:05:31.326823 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:31.326916 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":413,"threads_started":1}
I20260929 01:05:31.380168 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:31.379264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:60607)
0929 01:05:31.379840 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:31.379958 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":314,"threads_started":1}
I20260929 01:05:31.851912 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:31.851107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:55577)
0929 01:05:31.851642 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:31.851730 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":327,"threads_started":1}
I20260929 01:05:32.329481 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:32.328426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:45673)
0929 01:05:32.329077 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:32.329248 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":272,"threads_started":1}
I20260929 01:05:32.382499 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:32.381522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:51951)
0929 01:05:32.382141 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:32.382280 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":309,"threads_started":1}
I20260929 01:05:32.853900 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:32.852959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:34541)
0929 01:05:32.853472 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:32.853581 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260929 01:05:33.331524 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:33.330731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:38385)
0929 01:05:33.331269 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:33.331350 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":308,"threads_started":1}
I20260929 01:05:33.384604 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:33.383715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:42125)
0929 01:05:33.384262 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:33.384413 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":288,"threads_started":1}
I20260929 01:05:33.856137 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:33.855340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:54871)
0929 01:05:33.855845 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:33.855966 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
I20260929 01:05:34.333981 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:34.332923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:33797)
0929 01:05:34.333586 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:34.333730 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":337,"threads_started":1}
I20260929 01:05:34.386606 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:34.385790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:49257)
0929 01:05:34.386288 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:34.386376 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
I20260929 01:05:34.858349 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:34.857351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:49597)
0929 01:05:34.858042 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:34.858148 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":326,"threads_started":1}
I20260929 01:05:35.336006 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:35.335183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:40593)
0929 01:05:35.335707 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:35.335797 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
I20260929 01:05:35.389073 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:35.387848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:59273)
0929 01:05:35.388586 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:35.388712 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":471,"threads_started":1}
I20260929 01:05:35.860766 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:35.859654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:41439)
0929 01:05:35.860465 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:35.860554 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":548,"threads_started":1}
I20260929 01:05:36.338229 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:36.337354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:59479)
0929 01:05:36.337829 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:36.338031 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260929 01:05:36.391088 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:36.390197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:56969)
0929 01:05:36.390729 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:36.390895 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":309,"threads_started":1}
I20260929 01:05:36.863107 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:36.862054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.194:39167)
0929 01:05:36.862772 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:36.862904 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":372,"threads_started":1}
I20260929 01:05:37.340368 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:37.339518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.193:52261)
0929 01:05:37.340031 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:37.340127 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260929 01:05:37.370473 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19814
I20260929 01:05:37.393394 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:37.392576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34335 (local address 127.3.21.195:49889)
0929 01:05:37.393085 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:37.393199 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.254:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
I20260929 01:05:37.400187 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:37.372284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.3.21.193:43597)
0929 01:05:37.375232 (+ 2948us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:37.375246 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:37.375294 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:05:37.377012 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:37.377022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:37.377496 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:37.377833 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:37.379631 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:37.381022 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:37.382509 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:37.383321 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:37.383497 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:37.392766 (+ 9269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:37.392778 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:37.392798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:37.392847 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:37.396418 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:37.397274 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:37.397282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:37.397289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:37.397506 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:37.397933 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:37.397938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:37.397940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:37.398656 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:37.398882 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:37.399358 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 01:05:37.399877 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2765,"thread_start_us":2593,"threads_started":1}
I20260929 01:05:37.401749 19977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:37.489250 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19814
I20260929 01:05:37.507498 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 19985
I20260929 01:05:37.527966 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:37.508563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.3.21.194:46499)
0929 01:05:37.509100 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:37.509114 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:37.509150 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:05:37.511009 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:37.511020 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:37.511459 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:37.511828 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:37.513407 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:37.514799 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:37.516302 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:37.517018 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:37.517237 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:37.520288 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:37.520297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:37.520317 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:37.520366 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:37.524102 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:37.524941 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:37.524959 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:37.524967 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:37.525166 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:37.525602 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:37.525608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:37.525612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:37.526233 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:37.526605 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:37.527155 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 01:05:37.527630 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
I20260929 01:05:37.529768 20108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:37.618148 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 19985
I20260929 01:05:37.636399 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20116
I20260929 01:05:37.657608 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:37.637492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.3.21.195:41277)
0929 01:05:37.638136 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:37.638150 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:37.638197 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:05:37.639948 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:37.639958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:37.640444 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:37.640819 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:37.642343 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:37.643684 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:37.645034 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:37.645734 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:37.645956 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:37.649071 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:37.649081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:37.649099 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:37.649150 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:37.653251 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:37.654230 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:37.654240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:37.654251 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:37.654461 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:37.654974 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:37.654981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:37.654985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:37.655688 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:37.656048 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:37.656617 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 01:05:37.657246 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":324,"threads_started":1}
I20260929 01:05:37.659250 20240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:37.751771 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20116
2026-09-29T01:05:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65409 ms)
[----------] 30 tests from SecurityITest (325861 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:05:37 dist-test-slave-k0b7 krb5kdc[20302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:05:37 dist-test-slave-k0b7 krb5kdc[20302](info): set up 2 sockets
Sep 29 01:05:37 dist-test-slave-k0b7 krb5kdc[20302](info): commencing operation
krb5kdc: starting...
W20260929 01:05:38.915264 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:05:39 dist-test-slave-k0b7 krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:05:39.199988 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:43455
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:43455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:39.564663 20315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:39.564997 20315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:39.565107 20315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:39.572875 20315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:39.573019 20315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:39.573110 20315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:39.573201 20315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:39.573309 20315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:39.584606 20315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:43455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:43455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:39.586057 20315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:39.587548 20315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:39.602180 20321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:39.602085 20323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:39.602237 20320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:39.602921 20315 server_base.cc:1034] running on GCE node
I20260929 01:05:39.604126 20315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:39.605799 20315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:39.607082 20315 hybrid_clock.cc:648] HybridClock initialized: now 1790643939607021 us; error 48 us; skew 500 ppm
I20260929 01:05:39.607534 20315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:39 dist-test-slave-k0b7 krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643939, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:39.611279 20315 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:05:39.613356 20315 webserver.cc:460] Webserver started at http://127.3.21.254:40207/ using document root <none> and password file <none>
I20260929 01:05:39.614337 20315 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:39.614571 20315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:39.614951 20315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:39.618765 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "dfae8b0506224ea08792e82ff9e17c23"
format_stamp: "Formatted at 2026-09-29 01:05:39 on dist-test-slave-k0b7"
server_key: "118f91eb278b2fd4599b4e8dd1a36a34"
server_key_iv: "7356b0ef4d8fa111900319c56d91ca12"
server_key_version: "encryptionkey@0"
I20260929 01:05:39.619721 20315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "dfae8b0506224ea08792e82ff9e17c23"
format_stamp: "Formatted at 2026-09-29 01:05:39 on dist-test-slave-k0b7"
server_key: "118f91eb278b2fd4599b4e8dd1a36a34"
server_key_iv: "7356b0ef4d8fa111900319c56d91ca12"
server_key_version: "encryptionkey@0"
I20260929 01:05:39.625996 20315 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 01:05:39.630296 20330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:39.632193 20315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:39.632434 20315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "dfae8b0506224ea08792e82ff9e17c23"
format_stamp: "Formatted at 2026-09-29 01:05:39 on dist-test-slave-k0b7"
server_key: "118f91eb278b2fd4599b4e8dd1a36a34"
server_key_iv: "7356b0ef4d8fa111900319c56d91ca12"
server_key_version: "encryptionkey@0"
I20260929 01:05:39.632674 20315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:39.648945 20315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:39.654419 20315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:39.654798 20315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:39.678268 20315 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:43455
I20260929 01:05:39.678290 20381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:43455 every 8 connection(s)
I20260929 01:05:39.679996 20315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:39.684266 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:39.688206 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20315
I20260929 01:05:39.688431 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:05:39.689327 3159 external_mini_cluster.cc:1428] Setting key 3ba5bbc10da105fe73b164a7fb89401e
I20260929 01:05:39.699445 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23: Bootstrap starting.
I20260929 01:05:39.703856 20382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:39.705320 20382 log.cc:824] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:05:39 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:39.708811 20382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23: No bootstrap required, opened a new log
I20260929 01:05:39.716436 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:39.691822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.3.21.254:43455)
0929 01:05:39.692725 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:39.692738 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:39.692785 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:05:39.694758 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:39.694787 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:39.695344 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:39.695748 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:39.697730 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:39.699417 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:39.701555 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:39.702447 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:39.702695 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:39.708329 (+ 5634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:39.708338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:39.708361 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:39.708403 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:39.711810 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:39.712643 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:39.712650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:39.712657 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:39.712827 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:39.713243 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:39.713248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:39.713253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:39.713863 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:39.714195 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:39.714681 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 01:05:39.715157 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":440,"threads_started":1}
I20260929 01:05:39.717320 20382 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfae8b0506224ea08792e82ff9e17c23" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 43455 } }
I20260929 01:05:39.717720 20382 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:39.717844 20382 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfae8b0506224ea08792e82ff9e17c23, State: Initialized, Role: FOLLOWER
I20260929 01:05:39.718564 20382 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfae8b0506224ea08792e82ff9e17c23" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 43455 } }
I20260929 01:05:39.718823 20382 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:39.718990 20382 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:39.719180 20382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:39.721666 20382 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfae8b0506224ea08792e82ff9e17c23" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 43455 } }
I20260929 01:05:39.722373 20382 leader_election.cc:302] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfae8b0506224ea08792e82ff9e17c23; no voters:
I20260929 01:05:39.723186 20382 leader_election.cc:288] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:39.723349 20387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:39.724088 20387 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [term 1 LEADER]: Becoming Leader. State: Replica: dfae8b0506224ea08792e82ff9e17c23, State: Running, Role: LEADER
I20260929 01:05:39.724774 20387 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfae8b0506224ea08792e82ff9e17c23" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 43455 } }
I20260929 01:05:39.725612 20382 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:39.728549 20388 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfae8b0506224ea08792e82ff9e17c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfae8b0506224ea08792e82ff9e17c23" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 43455 } } }
I20260929 01:05:39.728513 20389 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfae8b0506224ea08792e82ff9e17c23. Latest consensus state: current_term: 1 leader_uuid: "dfae8b0506224ea08792e82ff9e17c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfae8b0506224ea08792e82ff9e17c23" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 43455 } } }
I20260929 01:05:39.729054 20388 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:39.729076 20389 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:39.734048 20395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:39.742063 20395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:39.752516 20395 catalog_manager.cc:1347] Generated new cluster ID: 37da0ce12cea4cc68bdd4b19e3715b2d
I20260929 01:05:39.752658 20395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:39.779103 20395 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:39.781317 20395 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:39.797627 20395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23: Generated new TSK 0
I20260929 01:05:39.798739 20395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:05:39.891582 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:43455
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:40.275283 20408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:40.275666 20408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:40.275851 20408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:40.284057 20408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:40.284288 20408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:40.284518 20408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:40.296424 20408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:43455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:40.298231 20408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:40.299839 20408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:40.315349 20413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:40.315303 20416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:40.315320 20414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:40.317162 20408 server_base.cc:1034] running on GCE node
I20260929 01:05:40.317927 20408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:40.319100 20408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:40.320416 20408 hybrid_clock.cc:648] HybridClock initialized: now 1790643940320298 us; error 56 us; skew 500 ppm
I20260929 01:05:40.320771 20408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:40 dist-test-slave-k0b7 krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643940, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:40.324731 20408 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:05:40.326735 20408 webserver.cc:460] Webserver started at http://127.3.21.193:43681/ using document root <none> and password file <none>
I20260929 01:05:40.327585 20408 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:40.327756 20408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:40.328127 20408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:40.331864 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7"
format_stamp: "Formatted at 2026-09-29 01:05:40 on dist-test-slave-k0b7"
server_key: "da3844e57a653271ee6e8aee8c667a9b"
server_key_iv: "a90b50006320b2f5aa81a48afcbe1597"
server_key_version: "encryptionkey@0"
I20260929 01:05:40.332813 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7"
format_stamp: "Formatted at 2026-09-29 01:05:40 on dist-test-slave-k0b7"
server_key: "da3844e57a653271ee6e8aee8c667a9b"
server_key_iv: "a90b50006320b2f5aa81a48afcbe1597"
server_key_version: "encryptionkey@0"
I20260929 01:05:40.339620 20408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:05:40.343693 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:40.345717 20408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:05:40.345889 20408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7"
format_stamp: "Formatted at 2026-09-29 01:05:40 on dist-test-slave-k0b7"
server_key: "da3844e57a653271ee6e8aee8c667a9b"
server_key_iv: "a90b50006320b2f5aa81a48afcbe1597"
server_key_version: "encryptionkey@0"
I20260929 01:05:40.346119 20408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:40.363735 20408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:40.369010 20408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:40.369289 20408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:40.370316 20408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:40.371924 20408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:40.372002 20408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:40.372097 20408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:40.372136 20408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:40.411716 20408 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:42583
I20260929 01:05:40.411769 20535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:42583 every 8 connection(s)
I20260929 01:05:40.413197 20408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:40.416564 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20408
I20260929 01:05:40.416833 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:40.417593 3159 external_mini_cluster.cc:1428] Setting key f0126ecf504f185bc444a0c4a64c50b1
Sep 29 01:05:40 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643940, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:40.440361 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:40.416423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:49619 (local address 127.3.21.254:43455)
0929 01:05:40.417156 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:40.417165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:40.418190 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 01:05:40.420833 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:40.420841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:40.420939 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:40.421059 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:40.423117 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:40.424325 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:40.426534 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:40.427281 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:40.427441 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:40.431901 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:40.431910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:40.431918 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:40.431978 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:40.436155 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:40.437258 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:40.437267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:40.437274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:40.437449 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:40.438050 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:40.438058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:40.438064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:40.438462 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:40.438662 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:40.439557 (+ 895us) server_negotiation.cc:299] Negotiation successful
0929 01:05:40.439839 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":280,"threads_started":1}
I20260929 01:05:40.441062 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:40.417002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:43455 (local address 127.3.21.193:49619)
0929 01:05:40.417987 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:40.418037 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:40.420501 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:40.421399 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:40.421409 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:40.422106 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:40.422903 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:40.422927 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:40.424546 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:40.424551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:40.426316 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:40.426333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:40.427544 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:40.427550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:40.427914 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:40.428807 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:40.428833 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:40.431603 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:40.436386 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:40.436395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:40.436411 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:40.437084 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:40.437646 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:40.437650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:40.437652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:40.437882 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:40.438633 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:40.438640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:40.439132 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 01:05:40.439385 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 01:05:40.439828 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":426,"threads_started":1}
I20260929 01:05:40.443143 20536 heartbeater.cc:347] Connected to a master server at 127.3.21.254:43455
I20260929 01:05:40.443516 20536 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:40.444557 20536 heartbeater.cc:511] Master 127.3.21.254:43455 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:05:40.447594 20347 ts_manager.cc:194] Registered new tserver with Master: c4f6365a491845cfbaaeb78f2d8a5cb7 (127.3.21.193:42583)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:05:40.449585 20347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:49619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:05:40.487784 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:43455
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:40.846012 20542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:40.846478 20542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:40.846688 20542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:40.854435 20542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:40.854624 20542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:40.854802 20542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:05:40.865103 20542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:43455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:40.866636 20542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:40.868002 20542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:40.887507 20547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:40.887435 20548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:40.887472 20550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:40.888453 20542 server_base.cc:1034] running on GCE node
I20260929 01:05:40.889181 20542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:40.890338 20542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:40.891623 20542 hybrid_clock.cc:648] HybridClock initialized: now 1790643940891575 us; error 51 us; skew 500 ppm
I20260929 01:05:40.892032 20542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:40 dist-test-slave-k0b7 krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643940, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:40.895991 20542 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:05:40.897854 20542 webserver.cc:460] Webserver started at http://127.3.21.194:37723/ using document root <none> and password file <none>
I20260929 01:05:40.898808 20542 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:40.898974 20542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:40.899339 20542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:40.902844 20542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "5ca8ac7f024d49f2861975a4b090144a"
format_stamp: "Formatted at 2026-09-29 01:05:40 on dist-test-slave-k0b7"
server_key: "d5a0146287c54ee8712c243e9b523f04"
server_key_iv: "4ed25aed9506baa81368b555c158f123"
server_key_version: "encryptionkey@0"
I20260929 01:05:40.903831 20542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "5ca8ac7f024d49f2861975a4b090144a"
format_stamp: "Formatted at 2026-09-29 01:05:40 on dist-test-slave-k0b7"
server_key: "d5a0146287c54ee8712c243e9b523f04"
server_key_iv: "4ed25aed9506baa81368b555c158f123"
server_key_version: "encryptionkey@0"
I20260929 01:05:40.910097 20542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:05:40.914548 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:40.916179 20542 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:40.916392 20542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "5ca8ac7f024d49f2861975a4b090144a"
format_stamp: "Formatted at 2026-09-29 01:05:40 on dist-test-slave-k0b7"
server_key: "d5a0146287c54ee8712c243e9b523f04"
server_key_iv: "4ed25aed9506baa81368b555c158f123"
server_key_version: "encryptionkey@0"
I20260929 01:05:40.916651 20542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:40.955559 20542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:40.960628 20542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:40.960994 20542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:40.962176 20542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:40.963703 20542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:40.963814 20542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:40.963981 20542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:40.964071 20542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:41.003401 20542 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:40477
I20260929 01:05:41.003437 20669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:40477 every 8 connection(s)
I20260929 01:05:41.004853 20542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:41.006222 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20542
I20260929 01:05:41.006438 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:05:41.006996 3159 external_mini_cluster.cc:1428] Setting key ff8a3e48adef64c25b060e14b178152e
Sep 29 01:05:41 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643940, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:05:41.039130 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.008028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:57157 (local address 127.3.21.254:43455)
0929 01:05:41.015541 (+ 7513us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.015551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.016010 (+ 459us) server_negotiation.cc:407] Connection header received
0929 01:05:41.018556 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.018561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.018669 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.018800 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:41.021578 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.022771 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.025613 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.026308 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.026427 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.031794 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:41.031800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:41.031804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:41.031886 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:41.035104 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:41.036124 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:41.036135 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:41.036138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:41.036306 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:41.037083 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:41.037087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:41.037090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:41.037407 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:41.037544 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.038652 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.038848 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7352,"thread_start_us":7245,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:05:41.040402 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.009025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:43455 (local address 127.3.21.194:57157)
0929 01:05:41.015742 (+ 6717us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.015802 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.018181 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.019107 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.019122 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.020261 (+ 1139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:41.021300 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.021333 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.022969 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.022977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.025352 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.025376 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.026619 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.026630 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.027252 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.028304 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:41.028343 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:41.031466 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:41.035273 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:41.035284 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:41.035355 (+ 71us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:41.035926 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:41.036588 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:41.036598 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:41.036629 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:41.036907 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:41.037585 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:41.037591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:41.038138 (+ 547us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.038409 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.038940 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5853,"thread_start_us":5560,"threads_started":1}
I20260929 01:05:41.042217 20670 heartbeater.cc:347] Connected to a master server at 127.3.21.254:43455
I20260929 01:05:41.042685 20670 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:41.043543 20670 heartbeater.cc:511] Master 127.3.21.254:43455 requested a full tablet report, sending...
I20260929 01:05:41.045758 20347 ts_manager.cc:194] Registered new tserver with Master: 5ca8ac7f024d49f2861975a4b090144a (127.3.21.194:40477)
I20260929 01:05:41.046833 20347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:57157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:05:41.075855 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:43455
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:41.444932 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:41.445305 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:41.445487 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:41.453521 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:41.453715 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:41.453922 20676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:05:41.454556 20536 heartbeater.cc:503] Master 127.3.21.254:43455 was elected leader, sending a full tablet report...
I20260929 01:05:41.464900 20676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:43455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:41.466271 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:41.467685 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:41.482896 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:41.483034 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:41.482952 20684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:41.483726 20676 server_base.cc:1034] running on GCE node
I20260929 01:05:41.484426 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:41.485533 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:41.486800 20676 hybrid_clock.cc:648] HybridClock initialized: now 1790643941486739 us; error 48 us; skew 500 ppm
I20260929 01:05:41.487213 20676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:41 dist-test-slave-k0b7 krb5kdc[20302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643941, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:41.491041 20676 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:05:41.492800 20676 webserver.cc:460] Webserver started at http://127.3.21.195:45153/ using document root <none> and password file <none>
I20260929 01:05:41.493557 20676 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:41.493708 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:41.494035 20676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:41.497287 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c"
format_stamp: "Formatted at 2026-09-29 01:05:41 on dist-test-slave-k0b7"
server_key: "b04990904a71b5d26b7a08fc5b5b6d5a"
server_key_iv: "f68eddea2d3b7415278366c397c0e38b"
server_key_version: "encryptionkey@0"
I20260929 01:05:41.498221 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c"
format_stamp: "Formatted at 2026-09-29 01:05:41 on dist-test-slave-k0b7"
server_key: "b04990904a71b5d26b7a08fc5b5b6d5a"
server_key_iv: "f68eddea2d3b7415278366c397c0e38b"
server_key_version: "encryptionkey@0"
I20260929 01:05:41.504242 20676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 01:05:41.508628 20691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:41.510430 20676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 01:05:41.510748 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c"
format_stamp: "Formatted at 2026-09-29 01:05:41 on dist-test-slave-k0b7"
server_key: "b04990904a71b5d26b7a08fc5b5b6d5a"
server_key_iv: "f68eddea2d3b7415278366c397c0e38b"
server_key_version: "encryptionkey@0"
I20260929 01:05:41.511003 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:41.540202 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:41.545001 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:41.545328 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:41.546804 20676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:41.548594 20676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:41.548734 20676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:41.548931 20676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:41.549015 20676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:41.589651 20676 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:34747
I20260929 01:05:41.589709 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:34747 every 8 connection(s)
I20260929 01:05:41.591224 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:41.594971 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20676
I20260929 01:05:41.595212 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:05:41.595806 3159 external_mini_cluster.cc:1428] Setting key 9a63baba605b9ff8415022d671714770
Sep 29 01:05:41 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643941, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:41.614696 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.594342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:50221 (local address 127.3.21.254:43455)
0929 01:05:41.594981 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.594990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.598256 (+ 3266us) server_negotiation.cc:407] Connection header received
0929 01:05:41.598314 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.598318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.598441 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.598603 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:41.600549 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.601332 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.602986 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.603396 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.603532 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.607947 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:41.607953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:41.607957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:41.607995 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:41.611220 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:41.612071 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:41.612084 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:41.612087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:41.612216 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:41.612689 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:41.612701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:41.612704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:41.612992 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:41.613193 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.613996 (+ 803us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.614244 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260929 01:05:41.615805 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.594925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:43455 (local address 127.3.21.195:50221)
0929 01:05:41.595774 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.595828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.597911 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.598876 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.598885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.599658 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:41.600348 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.600367 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.601501 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.601505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.602821 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.602835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.603649 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.603656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.603934 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.604762 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:41.604790 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:41.607641 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:41.611392 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:41.611402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:41.611428 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:41.611925 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:41.612336 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:41.612341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:41.612345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:41.612560 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:41.613170 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:41.613178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:41.613559 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.613887 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.614562 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":378,"threads_started":1}
I20260929 01:05:41.617602 20804 heartbeater.cc:347] Connected to a master server at 127.3.21.254:43455
I20260929 01:05:41.618120 20804 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:41.619025 20804 heartbeater.cc:511] Master 127.3.21.254:43455 requested a full tablet report, sending...
I20260929 01:05:41.621186 20347 ts_manager.cc:194] Registered new tserver with Master: cdf3a6ed93b449e8b464ea3b6a03c57c (127.3.21.195:34747)
I20260929 01:05:41.622074 20347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:50221
I20260929 01:05:41.626600 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:41.647578 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.632437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.3.21.254:43455)
0929 01:05:41.632647 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.632651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.633091 (+ 440us) server_negotiation.cc:407] Connection header received
0929 01:05:41.633237 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.633241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.633335 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.633441 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:41.635233 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.636207 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.637757 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.638228 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.638354 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.641244 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:41.641250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:41.641254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:41.641301 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:41.644533 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:41.645290 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:41.645294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:41.645297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:41.645452 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:41.645998 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:41.646002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:41.646004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:41.646413 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:41.646721 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.647095 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.647286 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:05:41.652719 20347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:41.660094 20347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:41.686815 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.674679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:42583 (local address 127.0.0.1:39220)
0929 01:05:41.675595 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.675646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.675818 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.677616 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.677623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.677650 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:41.678085 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.678101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.680606 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.680613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.683243 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.683256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.685642 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.685653 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.685907 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.685969 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.686107 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.686338 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":1,"thread_start_us":495,"threads_started":1}
I20260929 01:05:41.687302 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.676433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.3.21.194:40477)
0929 01:05:41.678791 (+ 2358us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.678800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.678828 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:05:41.678915 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.678931 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.679168 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.680062 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:41.680283 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.681861 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.684574 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.686108 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.686348 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.686585 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.686803 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.686990 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2153,"thread_start_us":1980,"threads_started":1}
I20260929 01:05:41.687263 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.676031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:40477 (local address 127.0.0.1:40074)
0929 01:05:41.677310 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.677331 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.677451 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.679389 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.679394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.679420 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:41.679866 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.679877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.682069 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.682075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.684344 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.684359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.686350 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.686357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.686583 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.686626 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.686759 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.686894 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":636,"threads_started":1}
I20260929 01:05:41.688002 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.675012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.3.21.193:42583)
0929 01:05:41.676637 (+ 1625us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.676648 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.676683 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:05:41.676793 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.676804 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.677076 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.677275 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:41.678809 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.680438 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.683505 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.685074 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.685207 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.685483 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.687484 (+ 2001us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.687672 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447,"thread_start_us":1291,"threads_started":1}
I20260929 01:05:41.692286 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.678031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:34747 (local address 127.0.0.1:38192)
0929 01:05:41.679084 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.679110 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.679232 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.681561 (+ 2329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.681566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.681591 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:41.682097 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.682113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.685503 (+ 3390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.685511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.688929 (+ 3418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.688949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.691265 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.691272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.691454 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.691489 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.691588 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.691738 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":879,"thread_start_us":1273,"threads_started":1}
I20260929 01:05:41.692255 20605 tablet_service.cc:1467] Processing CreateTablet for tablet 7027c94b6fdf463bbc73b8915f17b4da (DEFAULT_TABLE table=test-table [id=6a31216a54e1479aae9dfbd8082e58a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:41.692328 20471 tablet_service.cc:1467] Processing CreateTablet for tablet 7027c94b6fdf463bbc73b8915f17b4da (DEFAULT_TABLE table=test-table [id=6a31216a54e1479aae9dfbd8082e58a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:41.693584 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.677506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.3.21.195:34747)
0929 01:05:41.680649 (+ 3143us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.680659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.680698 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:05:41.680796 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.680806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.681052 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.681234 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:41.682347 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.685285 (+ 2938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.689144 (+ 3859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.690724 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.690860 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.691102 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.693004 (+ 1902us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.693205 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":775,"threads_started":1}
I20260929 01:05:41.693893 20471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7027c94b6fdf463bbc73b8915f17b4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:41.693892 20605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7027c94b6fdf463bbc73b8915f17b4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:41.701083 20739 tablet_service.cc:1467] Processing CreateTablet for tablet 7027c94b6fdf463bbc73b8915f17b4da (DEFAULT_TABLE table=test-table [id=6a31216a54e1479aae9dfbd8082e58a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:41.702945 20739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7027c94b6fdf463bbc73b8915f17b4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:41.711329 20823 tablet_bootstrap.cc:492] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Bootstrap starting.
I20260929 01:05:41.715646 20823 tablet_bootstrap.cc:654] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:41.717144 20824 tablet_bootstrap.cc:492] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Bootstrap starting.
I20260929 01:05:41.717312 20823 log.cc:824] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:41.721110 20823 tablet_bootstrap.cc:492] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: No bootstrap required, opened a new log
I20260929 01:05:41.721359 20824 tablet_bootstrap.cc:654] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:41.721514 20823 ts_tablet_manager.cc:1397] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 01:05:41.721894 20826 tablet_bootstrap.cc:492] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Bootstrap starting.
I20260929 01:05:41.722884 20824 log.cc:824] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:41.725570 20826 tablet_bootstrap.cc:654] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:41.726599 20824 tablet_bootstrap.cc:492] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: No bootstrap required, opened a new log
I20260929 01:05:41.726892 20824 ts_tablet_manager.cc:1397] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260929 01:05:41.726944 20826 log.cc:824] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:41.730561 20826 tablet_bootstrap.cc:492] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: No bootstrap required, opened a new log
I20260929 01:05:41.730870 20826 ts_tablet_manager.cc:1397] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 01:05:41.732182 20823 raft_consensus.cc:348] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.732658 20823 raft_consensus.cc:374] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:41.732780 20823 raft_consensus.cc:729] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6365a491845cfbaaeb78f2d8a5cb7, State: Initialized, Role: FOLLOWER
I20260929 01:05:41.733507 20823 consensus_queue.cc:260] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.735774 20823 ts_tablet_manager.cc:1428] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260929 01:05:41.738834 20824 raft_consensus.cc:348] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.739549 20824 raft_consensus.cc:374] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:41.739711 20824 raft_consensus.cc:729] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca8ac7f024d49f2861975a4b090144a, State: Initialized, Role: FOLLOWER
I20260929 01:05:41.740482 20824 consensus_queue.cc:260] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.741783 20670 heartbeater.cc:503] Master 127.3.21.254:43455 was elected leader, sending a full tablet report...
I20260929 01:05:41.742092 20826 raft_consensus.cc:348] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.742589 20826 raft_consensus.cc:374] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:41.742699 20826 raft_consensus.cc:729] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf3a6ed93b449e8b464ea3b6a03c57c, State: Initialized, Role: FOLLOWER
I20260929 01:05:41.742784 20824 ts_tablet_manager.cc:1428] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 01:05:41.743348 20826 consensus_queue.cc:260] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.744503 20804 heartbeater.cc:503] Master 127.3.21.254:43455 was elected leader, sending a full tablet report...
I20260929 01:05:41.745159 20826 ts_tablet_manager.cc:1428] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260929 01:05:41.757121 20671 tablet.cc:2369] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:05:41.843472 20805 tablet.cc:2369] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:41.855530 20829 raft_consensus.cc:482] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:41.855820 20829 raft_consensus.cc:504] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.857661 20829 leader_election.cc:288] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ca8ac7f024d49f2861975a4b090144a (127.3.21.194:40477), cdf3a6ed93b449e8b464ea3b6a03c57c (127.3.21.195:34747)
I20260929 01:05:41.867726 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.858257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:57459 (local address 127.3.21.194:40477)
0929 01:05:41.858605 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.858610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.858975 (+ 365us) server_negotiation.cc:407] Connection header received
0929 01:05:41.859114 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.859119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.859204 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.859399 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:41.860184 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.861345 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.865058 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.866437 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.866620 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.866685 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.867339 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.867491 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260929 01:05:41.867699 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.859219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:39951 (local address 127.3.21.195:34747)
0929 01:05:41.859454 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.859461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.859722 (+ 261us) server_negotiation.cc:407] Connection header received
0929 01:05:41.859915 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.859919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.859986 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.860060 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:41.860584 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.861674 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.865058 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.866366 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.866604 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.866700 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.867239 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.867426 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:05:41.867884 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.858970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:34747 (local address 127.3.21.193:39951)
0929 01:05:41.859603 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.859625 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.859781 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.860104 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.860112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.860140 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:41.860420 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.860427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.861838 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.861844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.864818 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.864840 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.866815 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.866822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.867004 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.867049 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.867169 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.867345 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":102,"spinlock_wait_cycles":4224,"thread_start_us":321,"threads_started":1}
I20260929 01:05:41.867937 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.858049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:40477 (local address 127.3.21.193:57459)
0929 01:05:41.858800 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:41.858825 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:41.858975 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:41.859534 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:41.859539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:41.859565 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:41.860006 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.860016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.861539 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.861544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.864828 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:41.864840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.866965 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.866974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:41.867171 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.867201 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:05:41.867279 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 01:05:41.867391 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":174,"spinlock_wait_cycles":28544,"thread_start_us":393,"threads_started":1}
I20260929 01:05:41.869469 20759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027c94b6fdf463bbc73b8915f17b4da" candidate_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" is_pre_election: true
I20260929 01:05:41.869489 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027c94b6fdf463bbc73b8915f17b4da" candidate_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ca8ac7f024d49f2861975a4b090144a" is_pre_election: true
I20260929 01:05:41.870186 20625 raft_consensus.cc:2454] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6365a491845cfbaaeb78f2d8a5cb7 in term 0.
I20260929 01:05:41.870241 20759 raft_consensus.cc:2454] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6365a491845cfbaaeb78f2d8a5cb7 in term 0.
I20260929 01:05:41.871167 20425 leader_election.cc:302] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6365a491845cfbaaeb78f2d8a5cb7, cdf3a6ed93b449e8b464ea3b6a03c57c; no voters:
I20260929 01:05:41.871750 20829 raft_consensus.cc:2781] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:41.871989 20829 raft_consensus.cc:482] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:41.872145 20829 raft_consensus.cc:3037] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:41.874871 20829 raft_consensus.cc:504] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.875640 20829 leader_election.cc:288] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [CANDIDATE]: Term 1 election: Requested vote from peers 5ca8ac7f024d49f2861975a4b090144a (127.3.21.194:40477), cdf3a6ed93b449e8b464ea3b6a03c57c (127.3.21.195:34747)
I20260929 01:05:41.876595 20759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027c94b6fdf463bbc73b8915f17b4da" candidate_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c"
I20260929 01:05:41.876621 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027c94b6fdf463bbc73b8915f17b4da" candidate_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ca8ac7f024d49f2861975a4b090144a"
I20260929 01:05:41.876854 20759 raft_consensus.cc:3037] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:41.876964 20625 raft_consensus.cc:3037] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:41.879649 20759 raft_consensus.cc:2454] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6365a491845cfbaaeb78f2d8a5cb7 in term 1.
I20260929 01:05:41.879774 20625 raft_consensus.cc:2454] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6365a491845cfbaaeb78f2d8a5cb7 in term 1.
I20260929 01:05:41.880424 20425 leader_election.cc:302] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6365a491845cfbaaeb78f2d8a5cb7, cdf3a6ed93b449e8b464ea3b6a03c57c; no voters:
I20260929 01:05:41.881085 20829 raft_consensus.cc:2781] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:41.881736 20829 raft_consensus.cc:686] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6365a491845cfbaaeb78f2d8a5cb7, State: Running, Role: LEADER
I20260929 01:05:41.882390 20829 consensus_queue.cc:237] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } }
I20260929 01:05:41.888437 20347 catalog_manager.cc:5495] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6365a491845cfbaaeb78f2d8a5cb7 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6365a491845cfbaaeb78f2d8a5cb7" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 42583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 } health_report { overall_health: UNKNOWN } } }
W20260929 01:05:41.916514 20537 tablet.cc:2369] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:41.944485 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.937723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.3.21.193:42583)
0929 01:05:41.938017 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.938024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:41.938274 (+ 250us) server_negotiation.cc:407] Connection header received
0929 01:05:41.938588 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:41.938593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:41.938685 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:41.938877 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:41.939344 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.940125 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.941819 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:41.942284 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:41.942470 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:41.943159 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:41.943735 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:41.943922 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:41.944035 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 01:05:41.944138 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 01:05:41.953903 20625 raft_consensus.cc:1261] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 1 FOLLOWER]: Refusing update from remote peer c4f6365a491845cfbaaeb78f2d8a5cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:41.954037 20759 raft_consensus.cc:1261] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 1 FOLLOWER]: Refusing update from remote peer c4f6365a491845cfbaaeb78f2d8a5cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:41.955191 20834 consensus_queue.cc:1035] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ca8ac7f024d49f2861975a4b090144a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 40477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:41.955672 20829 consensus_queue.cc:1035] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf3a6ed93b449e8b464ea3b6a03c57c" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:41.969987 20838 mvcc.cc:205] Tried to move back new op lower bound from 7334477586232061952 to 7334477585955962880. Current Snapshot: MvccSnapshot[applied={T|T < 7334477586232061952}]
I20260929 01:05:41.986299 20347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:41.986657 20347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:41.991144 20347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfae8b0506224ea08792e82ff9e17c23: Sending DeleteTablet for 3 replicas of tablet 7027c94b6fdf463bbc73b8915f17b4da
I20260929 01:05:41.992764 20471 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027c94b6fdf463bbc73b8915f17b4da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:41 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:39220
I20260929 01:05:41.992834 20739 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027c94b6fdf463bbc73b8915f17b4da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:41 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:38192
I20260929 01:05:41.992946 20605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027c94b6fdf463bbc73b8915f17b4da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:41 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:40074
I20260929 01:05:41.995182 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20408
I20260929 01:05:42.005811 20849 tablet_replica.cc:330] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: stopping tablet replica
I20260929 01:05:42.005811 20847 tablet_replica.cc:330] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: stopping tablet replica
I20260929 01:05:42.006965 20849 raft_consensus.cc:2229] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:42.007886 20849 raft_consensus.cc:2258] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:42.006386 20847 raft_consensus.cc:2229] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:42.008720 20847 raft_consensus.cc:2258] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:42.009457 20848 tablet_replica.cc:330] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: stopping tablet replica
I20260929 01:05:42.010077 20848 raft_consensus.cc:2229] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:42.011034 20848 raft_consensus.cc:2258] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:42.011993 20847 ts_tablet_manager.cc:1905] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:42.013698 20848 ts_tablet_manager.cc:1905] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:42.019742 20849 ts_tablet_manager.cc:1905] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:42.020527 20847 ts_tablet_manager.cc:1918] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:42.020736 20847 log.cc:1196] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/7027c94b6fdf463bbc73b8915f17b4da
I20260929 01:05:42.021354 20847 ts_tablet_manager.cc:1939] T 7027c94b6fdf463bbc73b8915f17b4da P cdf3a6ed93b449e8b464ea3b6a03c57c: Deleting consensus metadata
I20260929 01:05:42.023677 20332 catalog_manager.cc:4841] TS cdf3a6ed93b449e8b464ea3b6a03c57c (127.3.21.195:34747): tablet 7027c94b6fdf463bbc73b8915f17b4da (table test-table [id=6a31216a54e1479aae9dfbd8082e58a4]) successfully deleted
I20260929 01:05:42.024230 20848 ts_tablet_manager.cc:1918] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:42.024561 20848 log.cc:1196] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/7027c94b6fdf463bbc73b8915f17b4da
I20260929 01:05:42.025373 20848 ts_tablet_manager.cc:1939] T 7027c94b6fdf463bbc73b8915f17b4da P c4f6365a491845cfbaaeb78f2d8a5cb7: Deleting consensus metadata
I20260929 01:05:42.027707 20332 catalog_manager.cc:4841] TS c4f6365a491845cfbaaeb78f2d8a5cb7 (127.3.21.193:42583): tablet 7027c94b6fdf463bbc73b8915f17b4da (table test-table [id=6a31216a54e1479aae9dfbd8082e58a4]) successfully deleted
I20260929 01:05:42.033157 20849 ts_tablet_manager.cc:1918] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:42.033384 20849 log.cc:1196] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/7027c94b6fdf463bbc73b8915f17b4da
I20260929 01:05:42.034055 20849 ts_tablet_manager.cc:1939] T 7027c94b6fdf463bbc73b8915f17b4da P 5ca8ac7f024d49f2861975a4b090144a: Deleting consensus metadata
I20260929 01:05:42.036473 20332 catalog_manager.cc:4841] TS 5ca8ac7f024d49f2861975a4b090144a (127.3.21.194:40477): tablet 7027c94b6fdf463bbc73b8915f17b4da (table test-table [id=6a31216a54e1479aae9dfbd8082e58a4]) successfully deleted
Sep 29 01:05:42 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:05:42.048429 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:41.996672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.3.21.193:42583)
0929 01:05:41.996870 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:41.996876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:42.024833 (+ 27957us) server_negotiation.cc:407] Connection header received
0929 01:05:42.025096 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:42.025103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:42.025199 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:42.025333 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:42.030309 (+ 4976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:42.031751 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:42.034360 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:42.035263 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:42.035405 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:42.041928 (+ 6523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:42.041943 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:42.041958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:42.042010 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:42.045323 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:42.046139 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:42.046147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:42.046149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:42.046308 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:42.046769 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:42.046772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:42.046774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:42.047159 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:42.047490 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:42.047811 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 01:05:42.048081 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:05:42.049209 20531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:42.165000 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20408
I20260929 01:05:42.184298 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20542
Sep 29 01:05:42 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:05:42.203576 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:42.185137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.3.21.194:40477)
0929 01:05:42.185341 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:42.185348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:42.185448 (+ 100us) server_negotiation.cc:407] Connection header received
0929 01:05:42.185704 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:42.185709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:42.185775 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:42.185897 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:42.187499 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:42.188680 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:42.190112 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:42.190657 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:42.190799 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:42.196020 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:42.196042 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:42.196060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:42.196112 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:42.200153 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:42.201260 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:42.201269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:42.201271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:42.201434 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:42.201852 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:42.201858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:42.201861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:42.202271 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:42.202671 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:42.202955 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 01:05:42.203197 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:05:42.204276 20665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:42.300489 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20542
I20260929 01:05:42.321147 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20676
Sep 29 01:05:42 dist-test-slave-k0b7 krb5kdc[20302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:05:42.340476 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:42.322247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.3.21.195:34747)
0929 01:05:42.322450 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:42.322457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:42.322618 (+ 161us) server_negotiation.cc:407] Connection header received
0929 01:05:42.322865 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:42.322869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:42.323009 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:42.323124 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:42.324530 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:42.325315 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:42.326832 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:42.327223 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:42.327364 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:42.332621 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:42.332639 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:42.332656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:42.332700 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:42.336647 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:42.337724 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:42.337737 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:42.337740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:42.337930 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:42.338474 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:42.338477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:42.338480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:42.339066 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:42.339308 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:42.339722 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 01:05:42.340009 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:05:42.341305 20799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:42.440850 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20676
I20260929 01:05:42.459167 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20315
I20260929 01:05:42.460404 20377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:42.575724 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20315
2026-09-29T01:05:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4828 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T01:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:42Z Disabled control of system clock
I20260929 01:05:42.666425 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:42015
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:05:43.026021 20858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:43.026357 20858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:43.026569 20858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:43.034291 20858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:43.034428 20858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:43.034534 20858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:43.034638 20858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:43.034703 20858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:43.044509 20858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:42015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:43.045926 20858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:43.047974 20858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:43.061663 20863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:43.061625 20866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:43.061676 20864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:43.062693 20858 server_base.cc:1034] running on GCE node
I20260929 01:05:43.063522 20858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:43.065097 20858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:43.066358 20858 hybrid_clock.cc:648] HybridClock initialized: now 1790643943066323 us; error 42 us; skew 500 ppm
I20260929 01:05:43.066771 20858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:43.075307 20858 webserver.cc:460] Webserver started at http://127.3.21.254:36079/ using document root <none> and password file <none>
I20260929 01:05:43.076179 20858 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:43.076325 20858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:43.076680 20858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:43.079916 20858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "017096cae34c4103ac1bb059d28cde11"
format_stamp: "Formatted at 2026-09-29 01:05:43 on dist-test-slave-k0b7"
server_key: "2a42812f16efb741705ab2b1a67787c2"
server_key_iv: "9ee40ae524f74ef9f9781d5e1b5cf649"
server_key_version: "encryptionkey@0"
I20260929 01:05:43.080796 20858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "017096cae34c4103ac1bb059d28cde11"
format_stamp: "Formatted at 2026-09-29 01:05:43 on dist-test-slave-k0b7"
server_key: "2a42812f16efb741705ab2b1a67787c2"
server_key_iv: "9ee40ae524f74ef9f9781d5e1b5cf649"
server_key_version: "encryptionkey@0"
I20260929 01:05:43.086704 20858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:05:43.091426 20872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:43.093251 20858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:05:43.093493 20858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "017096cae34c4103ac1bb059d28cde11"
format_stamp: "Formatted at 2026-09-29 01:05:43 on dist-test-slave-k0b7"
server_key: "2a42812f16efb741705ab2b1a67787c2"
server_key_iv: "9ee40ae524f74ef9f9781d5e1b5cf649"
server_key_version: "encryptionkey@0"
I20260929 01:05:43.093755 20858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:43.115525 20858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:43.116499 20858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:43.116834 20858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:43.138267 20858 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:42015
I20260929 01:05:43.138293 20923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:42015 every 8 connection(s)
I20260929 01:05:43.139734 20858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:43.143815 20924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:43.149183 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20858
I20260929 01:05:43.149446 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:05:43.149995 3159 external_mini_cluster.cc:1428] Setting key 0068ab053cc59d6b5a70989b8c5dade8
I20260929 01:05:43.156272 20924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11: Bootstrap starting.
I20260929 01:05:43.160308 20924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:43.161564 20924 log.cc:824] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:43.165601 20924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11: No bootstrap required, opened a new log
I20260929 01:05:43.165692 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:43.152274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.3.21.254:42015)
0929 01:05:43.153406 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:43.153417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:43.153467 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:05:43.155353 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:43.155378 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:43.155883 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:43.156273 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:43.156965 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:43.158437 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:43.160509 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:43.161357 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:43.161500 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:43.162889 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:43.162902 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:43.162931 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:43.163716 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:43.163807 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:43.163829 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:43.163948 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:43.164255 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 01:05:43.164634 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":427,"threads_started":1}
I20260929 01:05:43.174780 20924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017096cae34c4103ac1bb059d28cde11" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42015 } }
I20260929 01:05:43.175194 20924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:43.175334 20924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017096cae34c4103ac1bb059d28cde11, State: Initialized, Role: FOLLOWER
I20260929 01:05:43.175987 20924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017096cae34c4103ac1bb059d28cde11" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42015 } }
I20260929 01:05:43.176251 20924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:43.176400 20924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:43.176579 20924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:43.178975 20924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017096cae34c4103ac1bb059d28cde11" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42015 } }
I20260929 01:05:43.179616 20924 leader_election.cc:302] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 017096cae34c4103ac1bb059d28cde11; no voters:
I20260929 01:05:43.180380 20924 leader_election.cc:288] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:43.180567 20929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:43.181376 20929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [term 1 LEADER]: Becoming Leader. State: Replica: 017096cae34c4103ac1bb059d28cde11, State: Running, Role: LEADER
I20260929 01:05:43.182029 20929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017096cae34c4103ac1bb059d28cde11" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42015 } }
I20260929 01:05:43.182623 20924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:43.185132 20930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "017096cae34c4103ac1bb059d28cde11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017096cae34c4103ac1bb059d28cde11" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42015 } } }
I20260929 01:05:43.185336 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 017096cae34c4103ac1bb059d28cde11. Latest consensus state: current_term: 1 leader_uuid: "017096cae34c4103ac1bb059d28cde11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017096cae34c4103ac1bb059d28cde11" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 42015 } } }
I20260929 01:05:43.185889 20930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:43.185983 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:43.188522 20934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:43.194128 20934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:43.205294 20934 catalog_manager.cc:1347] Generated new cluster ID: 052a8ded2bd24933a00458be29702c6f
I20260929 01:05:43.205444 20934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:43.257416 20934 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:43.258896 20934 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:43.272821 20934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11: Generated new TSK 0
I20260929 01:05:43.273986 20934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:05:43.283636 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42015
--builtin_ntp_servers=127.3.21.212:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:05:43.681288 20948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:43.681597 20948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:43.681749 20948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:43.690062 20948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:43.690182 20948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:43.690347 20948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:43.700595 20948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:43.701947 20948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:43.703853 20948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:43.718851 20953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:43.718813 20954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:43.718888 20956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:43.719587 20948 server_base.cc:1034] running on GCE node
I20260929 01:05:43.720302 20948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:43.721416 20948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:43.722674 20948 hybrid_clock.cc:648] HybridClock initialized: now 1790643943722615 us; error 47 us; skew 500 ppm
I20260929 01:05:43.723042 20948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:43.725883 20948 webserver.cc:460] Webserver started at http://127.3.21.193:42685/ using document root <none> and password file <none>
I20260929 01:05:43.726748 20948 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:43.726845 20948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:43.727138 20948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:43.730192 20948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "6284a5d25e834656864d2e55a70c132b"
format_stamp: "Formatted at 2026-09-29 01:05:43 on dist-test-slave-k0b7"
server_key: "72e57d3452ec234080ed825054d14c10"
server_key_iv: "a0f51a4b4028e4fb36f25c1698636877"
server_key_version: "encryptionkey@0"
I20260929 01:05:43.731120 20948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "6284a5d25e834656864d2e55a70c132b"
format_stamp: "Formatted at 2026-09-29 01:05:43 on dist-test-slave-k0b7"
server_key: "72e57d3452ec234080ed825054d14c10"
server_key_iv: "a0f51a4b4028e4fb36f25c1698636877"
server_key_version: "encryptionkey@0"
I20260929 01:05:43.736775 20948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:05:43.741271 20962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:43.743059 20948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:43.743295 20948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "6284a5d25e834656864d2e55a70c132b"
format_stamp: "Formatted at 2026-09-29 01:05:43 on dist-test-slave-k0b7"
server_key: "72e57d3452ec234080ed825054d14c10"
server_key_iv: "a0f51a4b4028e4fb36f25c1698636877"
server_key_version: "encryptionkey@0"
I20260929 01:05:43.743574 20948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:43.765098 20948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:43.766026 20948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:43.766359 20948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:43.767606 20948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:43.769203 20948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:43.769331 20948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:43.769470 20948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:43.769567 20948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:43.809023 20948 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:39633
I20260929 01:05:43.809123 21074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:39633 every 8 connection(s)
I20260929 01:05:43.810382 20948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:43.818130 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 20948
I20260929 01:05:43.818370 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:43.819099 3159 external_mini_cluster.cc:1428] Setting key 58cf571e78c6096aaac7a87a7efb663a
I20260929 01:05:43.822733 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42015
--builtin_ntp_servers=127.3.21.212:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:05:43.827333 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:43.814139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:47235 (local address 127.3.21.254:42015)
0929 01:05:43.814666 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:43.814673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:43.816151 (+ 1478us) server_negotiation.cc:407] Connection header received
0929 01:05:43.818322 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:43.818343 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:43.818452 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:43.818624 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:43.820184 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:43.821200 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:43.823346 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:43.824217 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:43.824391 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:43.826110 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:43.826120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:43.826129 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:43.826159 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:43.826217 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:43.826236 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:43.826337 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:43.826648 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 01:05:43.826797 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":218,"threads_started":1}
I20260929 01:05:43.828435 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:43.814628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42015 (local address 127.3.21.193:47235)
0929 01:05:43.815942 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:43.815993 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:43.817965 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:43.818946 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:43.818961 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:43.819114 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:43.819990 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:43.820012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:43.821448 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:43.821453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:43.823165 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:43.823180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:43.824564 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:43.824570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:43.824862 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:43.825664 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:43.825666 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:43.825713 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:43.825819 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:43.826347 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:43.826361 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:43.826371 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 01:05:43.826605 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 01:05:43.827060 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":432,"threads_started":1}
I20260929 01:05:43.830224 21075 heartbeater.cc:347] Connected to a master server at 127.3.21.254:42015
I20260929 01:05:43.830638 21075 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:43.831709 21075 heartbeater.cc:511] Master 127.3.21.254:42015 requested a full tablet report, sending...
I20260929 01:05:43.834928 20889 ts_manager.cc:194] Registered new tserver with Master: 6284a5d25e834656864d2e55a70c132b (127.3.21.193:39633)
I20260929 01:05:43.836511 20889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.193:47235
W20260929 01:05:44.185220 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:44.185534 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:44.185678 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:44.193673 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:44.193814 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:44.193998 21079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:05:44.204142 21079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:44.205538 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:44.207469 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:44.223232 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:44.223275 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:44.223171 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:44.225049 21079 server_base.cc:1034] running on GCE node
I20260929 01:05:44.225833 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:44.227160 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:44.228457 21079 hybrid_clock.cc:648] HybridClock initialized: now 1790643944228366 us; error 76 us; skew 500 ppm
I20260929 01:05:44.228861 21079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:44.232331 21079 webserver.cc:460] Webserver started at http://127.3.21.194:43227/ using document root <none> and password file <none>
I20260929 01:05:44.233217 21079 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:44.233331 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:44.233740 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:44.237416 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "a91da4f9d977479b803e4fbab176a39e"
format_stamp: "Formatted at 2026-09-29 01:05:44 on dist-test-slave-k0b7"
server_key: "2f299ac169b828d370d1ca68c12647ec"
server_key_iv: "02df65331a41ae57ec8f4e6a60d07c14"
server_key_version: "encryptionkey@0"
I20260929 01:05:44.238387 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "a91da4f9d977479b803e4fbab176a39e"
format_stamp: "Formatted at 2026-09-29 01:05:44 on dist-test-slave-k0b7"
server_key: "2f299ac169b828d370d1ca68c12647ec"
server_key_iv: "02df65331a41ae57ec8f4e6a60d07c14"
server_key_version: "encryptionkey@0"
I20260929 01:05:44.245263 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:05:44.249919 21093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:44.251858 21079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:44.252081 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "a91da4f9d977479b803e4fbab176a39e"
format_stamp: "Formatted at 2026-09-29 01:05:44 on dist-test-slave-k0b7"
server_key: "2f299ac169b828d370d1ca68c12647ec"
server_key_iv: "02df65331a41ae57ec8f4e6a60d07c14"
server_key_version: "encryptionkey@0"
I20260929 01:05:44.252349 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:44.277925 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:44.279078 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:44.279477 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:44.280683 21079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:44.282464 21079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:44.282684 21079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 01:05:44.282837 21079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:44.282943 21079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:44.325663 21079 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:41513
I20260929 01:05:44.325762 21205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:41513 every 8 connection(s)
I20260929 01:05:44.327181 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:44.334995 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21079
I20260929 01:05:44.335210 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:05:44.335745 3159 external_mini_cluster.cc:1428] Setting key 0503b0eb439202f95afbe042eb0c6dc6
I20260929 01:05:44.338958 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42015
--builtin_ntp_servers=127.3.21.212:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:05:44.341046 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.330200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:34501 (local address 127.3.21.254:42015)
0929 01:05:44.330792 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:44.330797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:44.331743 (+ 946us) server_negotiation.cc:407] Connection header received
0929 01:05:44.333677 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:44.333684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:44.333788 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:44.333913 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:44.335236 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.336131 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.337786 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.338234 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.338373 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.340124 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:44.340129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:44.340133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:44.340152 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:44.340191 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:44.340203 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:44.340307 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:44.340567 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 01:05:44.340701 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":357,"threads_started":1}
I20260929 01:05:44.342147 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.330645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42015 (local address 127.3.21.194:34501)
0929 01:05:44.331572 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:44.331610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:44.333368 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:44.334184 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:44.334193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:44.334292 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:44.335007 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.335025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.336295 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.336299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.337630 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.337643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.338517 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.338526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.338772 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.339734 (+ 962us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:44.339737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:44.339782 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:44.339880 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:44.340307 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:44.340322 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:44.340333 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:05:44.340541 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 01:05:44.341021 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":290,"threads_started":1}
I20260929 01:05:44.344177 21206 heartbeater.cc:347] Connected to a master server at 127.3.21.254:42015
I20260929 01:05:44.344584 21206 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:44.345777 21206 heartbeater.cc:511] Master 127.3.21.254:42015 requested a full tablet report, sending...
I20260929 01:05:44.348874 20889 ts_manager.cc:194] Registered new tserver with Master: a91da4f9d977479b803e4fbab176a39e (127.3.21.194:41513)
I20260929 01:05:44.350103 20889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.194:34501
W20260929 01:05:44.717769 21210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:44.718165 21210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:44.718338 21210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:44.726166 21210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:44.726346 21210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:44.726540 21210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:05:44.736351 21210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:42015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:44.737962 21210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:44.739965 21210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:44.755573 21215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:44.755475 21216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:44.755595 21218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:44.755985 21210 server_base.cc:1034] running on GCE node
I20260929 01:05:44.757083 21210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:44.758334 21210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:44.759634 21210 hybrid_clock.cc:648] HybridClock initialized: now 1790643944759542 us; error 140 us; skew 500 ppm
I20260929 01:05:44.760111 21210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:44.763312 21210 webserver.cc:460] Webserver started at http://127.3.21.195:44863/ using document root <none> and password file <none>
I20260929 01:05:44.764382 21210 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:44.764544 21210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:44.764981 21210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:44.769438 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "bde8043684574f3b8a2ab17d0d7cbb79"
format_stamp: "Formatted at 2026-09-29 01:05:44 on dist-test-slave-k0b7"
server_key: "388bf1653cb96ab3c96c803ab886c02b"
server_key_iv: "ffb4df0cb77ae565db269849f9f7d405"
server_key_version: "encryptionkey@0"
I20260929 01:05:44.770529 21210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "bde8043684574f3b8a2ab17d0d7cbb79"
format_stamp: "Formatted at 2026-09-29 01:05:44 on dist-test-slave-k0b7"
server_key: "388bf1653cb96ab3c96c803ab886c02b"
server_key_iv: "ffb4df0cb77ae565db269849f9f7d405"
server_key_version: "encryptionkey@0"
I20260929 01:05:44.776443 21210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:05:44.780815 21224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:44.782830 21210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:44.783046 21210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "bde8043684574f3b8a2ab17d0d7cbb79"
format_stamp: "Formatted at 2026-09-29 01:05:44 on dist-test-slave-k0b7"
server_key: "388bf1653cb96ab3c96c803ab886c02b"
server_key_iv: "ffb4df0cb77ae565db269849f9f7d405"
server_key_version: "encryptionkey@0"
I20260929 01:05:44.783378 21210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:44.812450 21210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:44.813556 21210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:44.813894 21210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:44.815184 21210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:44.816906 21210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:44.816999 21210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:44.817073 21210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:44.817131 21210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:44.840730 21075 heartbeater.cc:503] Master 127.3.21.254:42015 was elected leader, sending a full tablet report...
I20260929 01:05:44.855425 21210 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:33805
I20260929 01:05:44.855479 21336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:33805 every 8 connection(s)
I20260929 01:05:44.856868 21210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:44.862025 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21210
I20260929 01:05:44.862239 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:05:44.862885 3159 external_mini_cluster.cc:1428] Setting key 12a1db4f16934099e346aa1092acea01
I20260929 01:05:44.871924 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.859876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:33633 (local address 127.3.21.254:42015)
0929 01:05:44.860398 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:44.860403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:44.861428 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 01:05:44.864344 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:44.864350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:44.864457 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:44.864584 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:44.865811 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.866781 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.868585 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.869039 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.869211 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.870893 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:44.870899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:44.870902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:44.870930 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:44.870973 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:44.870986 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:44.871096 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:44.871361 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 01:05:44.871481 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
I20260929 01:05:44.873205 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.860282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:42015 (local address 127.3.21.195:33633)
0929 01:05:44.861252 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:44.861291 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:44.864005 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:44.864854 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:44.864864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:44.864983 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:44.865622 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.865640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.866956 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.866965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.868384 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.868403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.869327 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.869335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.869619 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.870440 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:44.870444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:44.870538 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:44.870653 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:44.871097 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:44.871112 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:44.871125 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:05:44.871352 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 01:05:44.871850 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":289,"threads_started":1}
I20260929 01:05:44.875602 21337 heartbeater.cc:347] Connected to a master server at 127.3.21.254:42015
I20260929 01:05:44.876040 21337 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:44.876994 21337 heartbeater.cc:511] Master 127.3.21.254:42015 requested a full tablet report, sending...
I20260929 01:05:44.878783 20889 ts_manager.cc:194] Registered new tserver with Master: bde8043684574f3b8a2ab17d0d7cbb79 (127.3.21.195:33805)
I20260929 01:05:44.879789 20889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.195:33633
I20260929 01:05:44.892743 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:44.905043 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.897533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.3.21.254:42015)
0929 01:05:44.897770 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:44.897776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:44.898217 (+ 441us) server_negotiation.cc:407] Connection header received
0929 01:05:44.898405 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:44.898437 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:44.898701 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:44.898801 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:44.899353 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.900248 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.902207 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.902695 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.902813 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.904156 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:44.904161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:44.904165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:44.904191 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:44.904245 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:44.904262 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:44.904380 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:44.904578 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 01:05:44.904674 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 01:05:44.909914 20889 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:44.918071 20889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:44.953976 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.934047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.3.21.195:33805)
0929 01:05:44.944724 (+ 10677us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:44.944734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:44.944778 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:05:44.944894 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:44.944903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:44.945261 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:44.945535 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:44.946270 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.947943 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.950871 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.952611 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.952767 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.953109 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:44.953400 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 01:05:44.953586 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10445,"thread_start_us":10225,"threads_started":1}
I20260929 01:05:44.954062 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.933474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:33805 (local address 127.0.0.1:44480)
0929 01:05:44.935423 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:44.935472 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:44.935686 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:44.945560 (+ 9874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:44.945567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:44.945598 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:44.946045 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.946067 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.948144 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.948148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.950583 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.950598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.952864 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.952872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.953115 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.953178 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 01:05:44.953332 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 01:05:44.953564 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"mutex_wait_us":3,"spinlock_wait_cycles":11136,"thread_start_us":2417,"threads_started":1}
I20260929 01:05:44.954710 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.933520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:41513 (local address 127.0.0.1:38476)
0929 01:05:44.943015 (+ 9495us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:44.943048 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:44.943236 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:44.946621 (+ 3385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:44.946625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:44.946652 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:44.947260 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.947278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.949295 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.949304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.951890 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.951917 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.953808 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.953815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.954036 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.954081 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:05:44.954158 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 01:05:44.954325 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9288,"mutex_wait_us":90,"thread_start_us":11576,"threads_started":1}
I20260929 01:05:44.955735 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.933541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.3.21.194:41513)
0929 01:05:44.945994 (+ 12453us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:44.946013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:44.946044 (+ 31us) server_negotiation.cc:407] Connection header received
0929 01:05:44.946148 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:44.946155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:44.946378 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:44.946621 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:44.947488 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.949153 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.952135 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.953561 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.954648 (+ 1087us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.955002 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:44.955100 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 01:05:44.955320 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11300,"thread_start_us":11162,"threads_started":1}
I20260929 01:05:44.956259 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.935037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:39633 (local address 127.0.0.1:49670)
0929 01:05:44.945394 (+ 10357us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:44.945415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:44.945649 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:44.946751 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:44.946759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:44.946786 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:44.947342 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.947352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.950404 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.950410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.952781 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:44.952796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.955311 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.955317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:44.955554 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.955589 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:05:44.955685 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:05:44.955868 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10212,"mutex_wait_us":107,"thread_start_us":7538,"threads_started":1}
I20260929 01:05:44.956369 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:44.935656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.3.21.193:39633)
0929 01:05:44.943991 (+ 8335us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:44.944007 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:44.945523 (+ 1516us) server_negotiation.cc:407] Connection header received
0929 01:05:44.945894 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:44.945903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:44.946163 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:44.946354 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:44.948568 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.950216 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.953029 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:44.954549 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:44.954684 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:44.955230 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:44.955795 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 01:05:44.955984 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8070,"thread_start_us":7909,"threads_started":1}
I20260929 01:05:44.959427 21272 tablet_service.cc:1467] Processing CreateTablet for tablet 83aef87cfc7549c3a03388f3c7633afc (DEFAULT_TABLE table=test-table [id=949ff409b9564e8aa170be3424009a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:44.959427 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 83aef87cfc7549c3a03388f3c7633afc (DEFAULT_TABLE table=test-table [id=949ff409b9564e8aa170be3424009a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:44.960392 21010 tablet_service.cc:1467] Processing CreateTablet for tablet 83aef87cfc7549c3a03388f3c7633afc (DEFAULT_TABLE table=test-table [id=949ff409b9564e8aa170be3424009a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:44.960990 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aef87cfc7549c3a03388f3c7633afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:44.960994 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aef87cfc7549c3a03388f3c7633afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:44.961650 21010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aef87cfc7549c3a03388f3c7633afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:44.976046 21356 tablet_bootstrap.cc:492] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Bootstrap starting.
I20260929 01:05:44.978473 21357 tablet_bootstrap.cc:492] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Bootstrap starting.
I20260929 01:05:44.980681 21356 tablet_bootstrap.cc:654] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:44.982363 21356 log.cc:824] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:44.982789 21357 tablet_bootstrap.cc:654] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:44.983230 21358 tablet_bootstrap.cc:492] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Bootstrap starting.
I20260929 01:05:44.984483 21357 log.cc:824] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:44.987257 21356 tablet_bootstrap.cc:492] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: No bootstrap required, opened a new log
I20260929 01:05:44.987547 21358 tablet_bootstrap.cc:654] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:44.987638 21356 ts_tablet_manager.cc:1397] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 01:05:44.988905 21358 log.cc:824] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:44.989432 21357 tablet_bootstrap.cc:492] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: No bootstrap required, opened a new log
I20260929 01:05:44.989797 21357 ts_tablet_manager.cc:1397] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 01:05:44.992343 21358 tablet_bootstrap.cc:492] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: No bootstrap required, opened a new log
I20260929 01:05:44.992630 21358 ts_tablet_manager.cc:1397] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 01:05:44.999169 21356 raft_consensus.cc:348] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:44.999169 21357 raft_consensus.cc:348] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:44.999706 21357 raft_consensus.cc:374] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:44.999891 21356 raft_consensus.cc:374] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:44.999904 21357 raft_consensus.cc:729] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91da4f9d977479b803e4fbab176a39e, State: Initialized, Role: FOLLOWER
I20260929 01:05:45.000010 21356 raft_consensus.cc:729] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6284a5d25e834656864d2e55a70c132b, State: Initialized, Role: FOLLOWER
I20260929 01:05:45.000654 21357 consensus_queue.cc:260] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.000790 21356 consensus_queue.cc:260] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.002091 21206 heartbeater.cc:503] Master 127.3.21.254:42015 was elected leader, sending a full tablet report...
I20260929 01:05:45.002737 21357 ts_tablet_manager.cc:1428] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 01:05:45.004537 21358 raft_consensus.cc:348] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.005064 21358 raft_consensus.cc:374] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:45.005182 21358 raft_consensus.cc:729] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde8043684574f3b8a2ab17d0d7cbb79, State: Initialized, Role: FOLLOWER
I20260929 01:05:45.005895 21358 consensus_queue.cc:260] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.007371 21337 heartbeater.cc:503] Master 127.3.21.254:42015 was elected leader, sending a full tablet report...
I20260929 01:05:45.007899 21358 ts_tablet_manager.cc:1428] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 01:05:45.010514 21356 ts_tablet_manager.cc:1428] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
W20260929 01:05:45.063796 21076 tablet.cc:2369] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:05:45.079702 21207 tablet.cc:2369] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:45.097395 21362 raft_consensus.cc:482] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:45.097805 21362 raft_consensus.cc:504] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.099766 21362 leader_election.cc:288] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde8043684574f3b8a2ab17d0d7cbb79 (127.3.21.195:33805), 6284a5d25e834656864d2e55a70c132b (127.3.21.193:39633)
W20260929 01:05:45.108721 21338 tablet.cc:2369] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:45.109898 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.100521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:39977 (local address 127.3.21.193:39633)
0929 01:05:45.100739 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:45.100747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:45.101466 (+ 719us) server_negotiation.cc:407] Connection header received
0929 01:05:45.101659 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:45.101664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:45.101751 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:45.101878 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:45.102851 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.103976 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.107306 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.108681 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.108931 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.109017 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:45.109482 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 01:05:45.109643 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:05:45.109984 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.100391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:58325 (local address 127.3.21.195:33805)
0929 01:05:45.100624 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:45.100630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:45.101423 (+ 793us) server_negotiation.cc:407] Connection header received
0929 01:05:45.101645 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:45.101650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:45.101739 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:45.101859 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:45.102834 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.104025 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.107306 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.108740 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.108960 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.109025 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:45.109600 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 01:05:45.109751 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:05:45.109999 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.100315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:39633 (local address 127.3.21.194:39977)
0929 01:05:45.101333 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:45.101358 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:45.101509 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:45.101969 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:45.101974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:45.102003 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:45.102566 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:45.102586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.104175 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.104180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:45.106969 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:45.106994 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.108990 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.108998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:45.109235 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.109279 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 01:05:45.109413 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 01:05:45.109587 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":244,"spinlock_wait_cycles":22912,"thread_start_us":507,"threads_started":1}
I20260929 01:05:45.110728 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.100292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:33805 (local address 127.3.21.194:58325)
0929 01:05:45.101253 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:45.101275 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:45.101471 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:45.102090 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:45.102096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:45.102138 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:45.102632 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:45.102640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.104175 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.104180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:45.106969 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:45.106994 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.109172 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.109178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:45.109356 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.109425 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 01:05:45.110157 (+ 732us) client_negotiation.cc:240] Negotiation successful
0929 01:05:45.110326 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":211,"thread_start_us":610,"threads_started":1}
I20260929 01:05:45.111253 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef87cfc7549c3a03388f3c7633afc" candidate_uuid: "a91da4f9d977479b803e4fbab176a39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6284a5d25e834656864d2e55a70c132b" is_pre_election: true
I20260929 01:05:45.111793 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef87cfc7549c3a03388f3c7633afc" candidate_uuid: "a91da4f9d977479b803e4fbab176a39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" is_pre_election: true
I20260929 01:05:45.112003 21030 raft_consensus.cc:2454] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91da4f9d977479b803e4fbab176a39e in term 0.
I20260929 01:05:45.112298 21292 raft_consensus.cc:2454] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a91da4f9d977479b803e4fbab176a39e in term 0.
I20260929 01:05:45.112923 21097 leader_election.cc:302] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6284a5d25e834656864d2e55a70c132b, a91da4f9d977479b803e4fbab176a39e; no voters:
I20260929 01:05:45.113474 21362 raft_consensus.cc:2781] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:45.113660 21362 raft_consensus.cc:482] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:45.113790 21362 raft_consensus.cc:3037] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:45.117347 21362 raft_consensus.cc:504] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.118072 21362 leader_election.cc:288] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [CANDIDATE]: Term 1 election: Requested vote from peers bde8043684574f3b8a2ab17d0d7cbb79 (127.3.21.195:33805), 6284a5d25e834656864d2e55a70c132b (127.3.21.193:39633)
I20260929 01:05:45.118881 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef87cfc7549c3a03388f3c7633afc" candidate_uuid: "a91da4f9d977479b803e4fbab176a39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde8043684574f3b8a2ab17d0d7cbb79"
I20260929 01:05:45.118860 21030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83aef87cfc7549c3a03388f3c7633afc" candidate_uuid: "a91da4f9d977479b803e4fbab176a39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6284a5d25e834656864d2e55a70c132b"
I20260929 01:05:45.119124 21292 raft_consensus.cc:3037] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:45.119124 21030 raft_consensus.cc:3037] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:45.122947 21030 raft_consensus.cc:2454] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91da4f9d977479b803e4fbab176a39e in term 1.
I20260929 01:05:45.122938 21292 raft_consensus.cc:2454] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a91da4f9d977479b803e4fbab176a39e in term 1.
I20260929 01:05:45.123569 21097 leader_election.cc:302] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6284a5d25e834656864d2e55a70c132b, a91da4f9d977479b803e4fbab176a39e; no voters:
I20260929 01:05:45.123951 21362 raft_consensus.cc:2781] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:45.124493 21362 raft_consensus.cc:686] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 1 LEADER]: Becoming Leader. State: Replica: a91da4f9d977479b803e4fbab176a39e, State: Running, Role: LEADER
I20260929 01:05:45.125061 21362 consensus_queue.cc:237] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } }
I20260929 01:05:45.131961 20889 catalog_manager.cc:5495] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e reported cstate change: term changed from 0 to 1, leader changed from <none> to a91da4f9d977479b803e4fbab176a39e (127.3.21.194). New cstate: current_term: 1 leader_uuid: "a91da4f9d977479b803e4fbab176a39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91da4f9d977479b803e4fbab176a39e" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 } health_report { overall_health: UNKNOWN } } }
I20260929 01:05:45.189378 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.183108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.3.21.194:41513)
0929 01:05:45.183329 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:45.183334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:45.183351 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:05:45.183444 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:45.183447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:45.183541 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:45.183685 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:45.184089 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.184844 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.186569 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.187053 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.187219 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.187845 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:45.188542 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:45.188682 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:45.188838 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 01:05:45.189067 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 01:05:45.198768 21292 raft_consensus.cc:1261] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 1 FOLLOWER]: Refusing update from remote peer a91da4f9d977479b803e4fbab176a39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:45.198792 21030 raft_consensus.cc:1261] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 1 FOLLOWER]: Refusing update from remote peer a91da4f9d977479b803e4fbab176a39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:45.200076 21367 consensus_queue.cc:1035] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde8043684574f3b8a2ab17d0d7cbb79" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:45.200477 21362 consensus_queue.cc:1035] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6284a5d25e834656864d2e55a70c132b" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:45.216758 21372 mvcc.cc:205] Tried to move back new op lower bound from 7334477599521906688 to 7334477599236710400. Current Snapshot: MvccSnapshot[applied={T|T < 7334477599521906688}]
I20260929 01:05:45.227828 20887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:45.228238 20887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:45.233351 20887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 017096cae34c4103ac1bb059d28cde11: Sending DeleteTablet for 3 replicas of tablet 83aef87cfc7549c3a03388f3c7633afc
I20260929 01:05:45.234920 21272 tablet_service.cc:1514] Processing DeleteTablet for tablet 83aef87cfc7549c3a03388f3c7633afc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:45 UTC) from {username='slave'} at 127.0.0.1:44480
I20260929 01:05:45.234977 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 83aef87cfc7549c3a03388f3c7633afc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:45 UTC) from {username='slave'} at 127.0.0.1:38476
I20260929 01:05:45.234937 21010 tablet_service.cc:1514] Processing DeleteTablet for tablet 83aef87cfc7549c3a03388f3c7633afc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:45 UTC) from {username='slave'} at 127.0.0.1:49670
I20260929 01:05:45.239208 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20948
I20260929 01:05:45.247917 21379 tablet_replica.cc:330] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: stopping tablet replica
I20260929 01:05:45.248262 21381 tablet_replica.cc:330] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: stopping tablet replica
I20260929 01:05:45.248477 21380 tablet_replica.cc:330] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: stopping tablet replica
I20260929 01:05:45.248564 21379 raft_consensus.cc:2229] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:45.249074 21380 raft_consensus.cc:2229] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:45.249364 21379 raft_consensus.cc:2258] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:45.249523 21381 raft_consensus.cc:2229] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:45.250114 21381 raft_consensus.cc:2258] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:45.250212 21380 raft_consensus.cc:2258] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:45.252401 21381 ts_tablet_manager.cc:1905] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:45.252419 21379 ts_tablet_manager.cc:1905] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:45.260202 21379 ts_tablet_manager.cc:1918] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:45.260213 21381 ts_tablet_manager.cc:1918] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:45.260416 21381 log.cc:1196] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/83aef87cfc7549c3a03388f3c7633afc
I20260929 01:05:45.260478 21379 log.cc:1196] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/83aef87cfc7549c3a03388f3c7633afc
I20260929 01:05:45.261047 21379 ts_tablet_manager.cc:1939] T 83aef87cfc7549c3a03388f3c7633afc P a91da4f9d977479b803e4fbab176a39e: Deleting consensus metadata
I20260929 01:05:45.261040 21381 ts_tablet_manager.cc:1939] T 83aef87cfc7549c3a03388f3c7633afc P 6284a5d25e834656864d2e55a70c132b: Deleting consensus metadata
I20260929 01:05:45.263114 20876 catalog_manager.cc:4841] TS 6284a5d25e834656864d2e55a70c132b (127.3.21.193:39633): tablet 83aef87cfc7549c3a03388f3c7633afc (table test-table [id=949ff409b9564e8aa170be3424009a28]) successfully deleted
I20260929 01:05:45.263535 20875 catalog_manager.cc:4841] TS a91da4f9d977479b803e4fbab176a39e (127.3.21.194:41513): tablet 83aef87cfc7549c3a03388f3c7633afc (table test-table [id=949ff409b9564e8aa170be3424009a28]) successfully deleted
I20260929 01:05:45.264345 21380 ts_tablet_manager.cc:1905] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:45.265866 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.240855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.3.21.193:39633)
0929 01:05:45.241087 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:45.241094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:45.254221 (+ 13127us) server_negotiation.cc:407] Connection header received
0929 01:05:45.254796 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:45.254803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:45.254925 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:45.255036 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:45.255769 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.257200 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.259745 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.260582 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.260711 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.264003 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:45.264015 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:45.264034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:45.264898 (+ 864us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:45.264968 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:45.264996 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:45.265132 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:45.265335 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 01:05:45.265478 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 01:05:45.266927 21070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:45.272641 21380 ts_tablet_manager.cc:1918] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:45.272858 21380 log.cc:1196] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/83aef87cfc7549c3a03388f3c7633afc
I20260929 01:05:45.273355 21380 ts_tablet_manager.cc:1939] T 83aef87cfc7549c3a03388f3c7633afc P bde8043684574f3b8a2ab17d0d7cbb79: Deleting consensus metadata
I20260929 01:05:45.275640 20873 catalog_manager.cc:4841] TS bde8043684574f3b8a2ab17d0d7cbb79 (127.3.21.195:33805): tablet 83aef87cfc7549c3a03388f3c7633afc (table test-table [id=949ff409b9564e8aa170be3424009a28]) successfully deleted
I20260929 01:05:45.373420 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20948
I20260929 01:05:45.393802 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21079
I20260929 01:05:45.403200 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.394822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.3.21.194:41513)
0929 01:05:45.394998 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:45.395003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:45.395081 (+ 78us) server_negotiation.cc:407] Connection header received
0929 01:05:45.395305 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:45.395310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:45.395415 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:45.395519 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:45.395992 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.396933 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.399110 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.399609 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.399722 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.401174 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:45.401184 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:45.401199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:45.401925 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:45.401984 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:45.402014 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:45.402112 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:45.402538 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 01:05:45.402653 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:05:45.404127 21201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:45.521660 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21079
I20260929 01:05:45.540802 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21210
I20260929 01:05:45.548131 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:45.541577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.3.21.195:33805)
0929 01:05:45.541782 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:45.541791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:45.541816 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:05:45.542055 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:45.542061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:45.542152 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:45.542275 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:45.542719 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.543527 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.544864 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:45.545269 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:45.545412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:45.546713 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:45.546721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:45.546738 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:45.547322 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:45.547372 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:45.547391 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:45.547538 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:45.547671 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 01:05:45.547788 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:05:45.549120 21332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:45.647589 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21210
I20260929 01:05:45.667174 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 20858
I20260929 01:05:45.668493 20919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:45.792285 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 20858
2026-09-29T01:05:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:05:45 dist-test-slave-k0b7 krb5kdc[21389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:05:45 dist-test-slave-k0b7 krb5kdc[21389](info): set up 2 sockets
Sep 29 01:05:45 dist-test-slave-k0b7 krb5kdc[21389](info): commencing operation
krb5kdc: starting...
W20260929 01:05:46.955070 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:05:47 dist-test-slave-k0b7 krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:05:47.255010 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:46427
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:46427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:47.629331 21402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:47.629673 21402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:47.629843 21402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:47.637630 21402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:47.637760 21402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:47.637848 21402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:47.637939 21402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:47.638005 21402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:47.649401 21402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:46427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:47.651057 21402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:47.652495 21402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:47.668359 21410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.668318 21408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.668392 21407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.669970 21402 server_base.cc:1034] running on GCE node
I20260929 01:05:47.671128 21402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:47.672977 21402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:47.674243 21402 hybrid_clock.cc:648] HybridClock initialized: now 1790643947674177 us; error 59 us; skew 500 ppm
I20260929 01:05:47.674724 21402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:47 dist-test-slave-k0b7 krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:47.678342 21402 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:05:47.680431 21402 webserver.cc:460] Webserver started at http://127.3.21.254:34585/ using document root <none> and password file <none>
I20260929 01:05:47.681264 21402 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:47.681433 21402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:47.681788 21402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:47.685218 21402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "9952bcfceb2940b785b4bc7987187235"
format_stamp: "Formatted at 2026-09-29 01:05:47 on dist-test-slave-k0b7"
server_key: "4f413f8100840a474fe9d8070353d245"
server_key_iv: "d08ce70a0544af3b32c4a16c015f397b"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.686096 21402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "9952bcfceb2940b785b4bc7987187235"
format_stamp: "Formatted at 2026-09-29 01:05:47 on dist-test-slave-k0b7"
server_key: "4f413f8100840a474fe9d8070353d245"
server_key_iv: "d08ce70a0544af3b32c4a16c015f397b"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.692600 21402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:05:47.697254 21417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.699611 21402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:05:47.699872 21402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "9952bcfceb2940b785b4bc7987187235"
format_stamp: "Formatted at 2026-09-29 01:05:47 on dist-test-slave-k0b7"
server_key: "4f413f8100840a474fe9d8070353d245"
server_key_iv: "d08ce70a0544af3b32c4a16c015f397b"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.700121 21402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:47.726830 21402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:47.731926 21402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:47.732268 21402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:47.752851 21402 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:46427
I20260929 01:05:47.752887 21468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:46427 every 8 connection(s)
I20260929 01:05:47.754362 21402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:47.758718 21469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:47.764679 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21402
I20260929 01:05:47.764909 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:05:47.765674 3159 external_mini_cluster.cc:1428] Setting key 656b15ab2aae206d65c3f22d2979f86f
I20260929 01:05:47.773234 21469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235: Bootstrap starting.
I20260929 01:05:47.778671 21469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:47.780165 21469 log.cc:824] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:05:47 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:47.784238 21469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235: No bootstrap required, opened a new log
I20260929 01:05:47.797020 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.767989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.3.21.254:46427)
0929 01:05:47.768945 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:47.768958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:47.769015 (+ 57us) server_negotiation.cc:407] Connection header received
0929 01:05:47.771252 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:47.771282 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:47.771752 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:47.772131 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:47.774381 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:47.776852 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:47.778510 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:47.779672 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:47.779910 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:47.785305 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:47.785317 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:47.785340 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:47.785396 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:47.790444 (+ 5048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:47.791621 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:47.791632 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:47.791642 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:47.791898 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:47.792507 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:47.792512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:47.792516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:47.793407 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:47.793884 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:47.794628 (+ 744us) server_negotiation.cc:299] Negotiation successful
0929 01:05:47.795265 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":524,"threads_started":1}
I20260929 01:05:47.796657 21469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9952bcfceb2940b785b4bc7987187235" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46427 } }
I20260929 01:05:47.797231 21469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:47.797341 21469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9952bcfceb2940b785b4bc7987187235, State: Initialized, Role: FOLLOWER
I20260929 01:05:47.798169 21469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9952bcfceb2940b785b4bc7987187235" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46427 } }
I20260929 01:05:47.798674 21469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:47.798879 21469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:47.799327 21469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:47.802543 21469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9952bcfceb2940b785b4bc7987187235" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46427 } }
I20260929 01:05:47.803391 21469 leader_election.cc:302] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9952bcfceb2940b785b4bc7987187235; no voters:
I20260929 01:05:47.804245 21469 leader_election.cc:288] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:47.804468 21474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:47.805691 21474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [term 1 LEADER]: Becoming Leader. State: Replica: 9952bcfceb2940b785b4bc7987187235, State: Running, Role: LEADER
I20260929 01:05:47.806285 21474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9952bcfceb2940b785b4bc7987187235" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46427 } }
I20260929 01:05:47.807004 21469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:47.810297 21475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9952bcfceb2940b785b4bc7987187235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9952bcfceb2940b785b4bc7987187235" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46427 } } }
I20260929 01:05:47.810693 21476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9952bcfceb2940b785b4bc7987187235. Latest consensus state: current_term: 1 leader_uuid: "9952bcfceb2940b785b4bc7987187235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9952bcfceb2940b785b4bc7987187235" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46427 } } }
I20260929 01:05:47.811053 21475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:47.811148 21476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:47.816277 21482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:47.821393 21482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:47.831120 21482 catalog_manager.cc:1347] Generated new cluster ID: 0c9b2f4b7feb498fb0021b41c8a695f3
I20260929 01:05:47.831383 21482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:47.839759 21482 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:47.841995 21482 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:47.858412 21482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235: Generated new TSK 0
I20260929 01:05:47.859855 21482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:05:47.933823 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46427
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:48.322394 21495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:48.322827 21495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:48.323029 21495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:48.331465 21495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:48.331602 21495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:48.331792 21495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:48.342240 21495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:48.343711 21495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:48.345139 21495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:48.360575 21501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:48.360699 21500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:48.361574 21503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:48.362006 21495 server_base.cc:1034] running on GCE node
I20260929 01:05:48.362799 21495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:48.363929 21495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:48.365227 21495 hybrid_clock.cc:648] HybridClock initialized: now 1790643948365191 us; error 39 us; skew 500 ppm
I20260929 01:05:48.365599 21495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:48 dist-test-slave-k0b7 krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:48.376302 21495 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:05:48.378393 21495 webserver.cc:460] Webserver started at http://127.3.21.193:37283/ using document root <none> and password file <none>
I20260929 01:05:48.379263 21495 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:48.379441 21495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:48.379823 21495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:48.383540 21495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "ac23f76ec91043a9a3c6fcc902a35056"
format_stamp: "Formatted at 2026-09-29 01:05:48 on dist-test-slave-k0b7"
server_key: "fec5ca939d84f44320a455a0b2989b9c"
server_key_iv: "6161c6cd27e2f6ddb22aaa4f0e6549ee"
server_key_version: "encryptionkey@0"
I20260929 01:05:48.384539 21495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "ac23f76ec91043a9a3c6fcc902a35056"
format_stamp: "Formatted at 2026-09-29 01:05:48 on dist-test-slave-k0b7"
server_key: "fec5ca939d84f44320a455a0b2989b9c"
server_key_iv: "6161c6cd27e2f6ddb22aaa4f0e6549ee"
server_key_version: "encryptionkey@0"
I20260929 01:05:48.391181 21495 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 01:05:48.395772 21510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:48.397586 21495 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:05:48.397837 21495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "ac23f76ec91043a9a3c6fcc902a35056"
format_stamp: "Formatted at 2026-09-29 01:05:48 on dist-test-slave-k0b7"
server_key: "fec5ca939d84f44320a455a0b2989b9c"
server_key_iv: "6161c6cd27e2f6ddb22aaa4f0e6549ee"
server_key_version: "encryptionkey@0"
I20260929 01:05:48.398213 21495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:48.417214 21495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:48.422984 21495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:48.423316 21495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:48.424506 21495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:48.426040 21495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:48.426126 21495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:48.426227 21495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:48.426280 21495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:48.466842 21495 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:36371
I20260929 01:05:48.466854 21622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:36371 every 8 connection(s)
I20260929 01:05:48.468415 21495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:48.474572 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21495
I20260929 01:05:48.474799 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:48.475289 3159 external_mini_cluster.cc:1428] Setting key d4efe0b9b7aede690a8e7f8a98b2b1b6
Sep 29 01:05:48 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:48.493613 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:48.471420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:45913 (local address 127.3.21.254:46427)
0929 01:05:48.472062 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:48.472068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:48.472821 (+ 753us) server_negotiation.cc:407] Connection header received
0929 01:05:48.474935 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:48.474941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:48.475048 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:48.475170 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:48.476999 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:48.477997 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:48.479715 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:48.480270 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:48.480421 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:48.485079 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:48.485086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:48.485089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:48.485126 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:48.489042 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:48.490595 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:48.490600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:48.490603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:48.490756 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:48.491405 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:48.491409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:48.491411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:48.491768 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:48.491932 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:48.493016 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0929 01:05:48.493246 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":285,"threads_started":1}
I20260929 01:05:48.495532 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:48.471872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:46427 (local address 127.3.21.193:45913)
0929 01:05:48.472633 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:48.472680 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:48.474611 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:48.475480 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:48.475489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:48.476202 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:48.476810 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:48.476828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:48.478179 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:48.478183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:48.479546 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:48.479560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:48.480555 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:48.480562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:48.480822 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:48.481667 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:48.481702 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:48.484748 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:48.489283 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:48.489297 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:48.489323 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:48.490286 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:48.490907 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:48.490915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:48.490922 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:48.491241 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:48.491928 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:48.491940 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:48.492491 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 01:05:48.492825 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 01:05:48.493485 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":297,"threads_started":1}
I20260929 01:05:48.497802 21623 heartbeater.cc:347] Connected to a master server at 127.3.21.254:46427
I20260929 01:05:48.498287 21623 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:48.499249 21623 heartbeater.cc:511] Master 127.3.21.254:46427 requested a full tablet report, sending...
I20260929 01:05:48.502041 21434 ts_manager.cc:194] Registered new tserver with Master: ac23f76ec91043a9a3c6fcc902a35056 (127.3.21.193:36371)
I20260929 01:05:48.503923 21434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:45913
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:05:48.544152 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46427
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:48.905242 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:48.905604 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:48.905781 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:48.914631 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:48.914800 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:48.914980 21629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:05:48.925518 21629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:48.927062 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:48.928438 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:48.943020 21635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:48.943171 21634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:48.943641 21637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:48.944051 21629 server_base.cc:1034] running on GCE node
I20260929 01:05:48.944684 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:48.945730 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:48.946986 21629 hybrid_clock.cc:648] HybridClock initialized: now 1790643948946878 us; error 74 us; skew 500 ppm
I20260929 01:05:48.947510 21629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:48 dist-test-slave-k0b7 krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:48.951556 21629 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:05:48.953454 21629 webserver.cc:460] Webserver started at http://127.3.21.194:32807/ using document root <none> and password file <none>
I20260929 01:05:48.954325 21629 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:48.954452 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:48.954964 21629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:48.959241 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "abd2bef33a174a8ab01015321c8bd2c2"
format_stamp: "Formatted at 2026-09-29 01:05:48 on dist-test-slave-k0b7"
server_key: "16365edfa1fc2695a9cf9ad9888b9187"
server_key_iv: "96d3585a6a748be13ea3ba3ded885049"
server_key_version: "encryptionkey@0"
I20260929 01:05:48.960155 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "abd2bef33a174a8ab01015321c8bd2c2"
format_stamp: "Formatted at 2026-09-29 01:05:48 on dist-test-slave-k0b7"
server_key: "16365edfa1fc2695a9cf9ad9888b9187"
server_key_iv: "96d3585a6a748be13ea3ba3ded885049"
server_key_version: "encryptionkey@0"
I20260929 01:05:48.966315 21629 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 01:05:48.971236 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:48.973114 21629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:05:48.973373 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "abd2bef33a174a8ab01015321c8bd2c2"
format_stamp: "Formatted at 2026-09-29 01:05:48 on dist-test-slave-k0b7"
server_key: "16365edfa1fc2695a9cf9ad9888b9187"
server_key_iv: "96d3585a6a748be13ea3ba3ded885049"
server_key_version: "encryptionkey@0"
I20260929 01:05:48.973663 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:49.008229 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:49.014204 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:49.014730 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:49.016341 21629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:49.018028 21629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:49.018181 21629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:49.018348 21629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:49.018433 21629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:49.061367 21629 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:34909
I20260929 01:05:49.061472 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:34909 every 8 connection(s)
I20260929 01:05:49.063237 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:49.063882 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21629
I20260929 01:05:49.064091 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:05:49.064697 3159 external_mini_cluster.cc:1428] Setting key 3c1c74f58bd60cbf83e5b0f3a2a1bbad
Sep 29 01:05:49 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:49.087689 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.066653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:40825 (local address 127.3.21.254:46427)
0929 01:05:49.067205 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.067212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.068452 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 01:05:49.070474 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.070502 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.070605 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.070745 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:49.072521 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.073287 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.075356 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.075820 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.076008 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.080434 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:49.080441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:49.080445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:49.080486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:49.083628 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.084567 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.084572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.084574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.084718 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.085209 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.085214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.085216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.085574 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:49.085749 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.086945 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.087224 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":234,"threads_started":1}
I20260929 01:05:49.088770 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.067352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:46427 (local address 127.3.21.194:40825)
0929 01:05:49.068196 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.068270 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.070168 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.071067 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.071077 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.071710 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:49.072316 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.072335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.073629 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.073633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.075164 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.075181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.076123 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.076130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.076388 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.077205 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:49.077229 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:49.079973 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:49.083823 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.083835 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.083865 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.084405 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.084845 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.084850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.084854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.085071 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.085722 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:49.085732 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:49.086422 (+ 690us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.086829 (+ 407us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.087453 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":361,"threads_started":1}
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:05:49.090763 21757 heartbeater.cc:347] Connected to a master server at 127.3.21.254:46427
I20260929 01:05:49.091147 21757 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:49.091897 21757 heartbeater.cc:511] Master 127.3.21.254:46427 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
I20260929 01:05:49.093864 21434 ts_manager.cc:194] Registered new tserver with Master: abd2bef33a174a8ab01015321c8bd2c2 (127.3.21.194:34909)
I20260929 01:05:49.094839 21434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:40825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:05:49.127381 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46427
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:49.480082 21763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:49.480391 21763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:49.480489 21763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:49.488185 21763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:49.488354 21763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:49.488511 21763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:05:49.499915 21763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:46427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:49.501374 21763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:49.502946 21763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:05:49.508047 21623 heartbeater.cc:503] Master 127.3.21.254:46427 was elected leader, sending a full tablet report...
W20260929 01:05:49.519229 21768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:49.519146 21769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:49.519677 21771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:49.520365 21763 server_base.cc:1034] running on GCE node
I20260929 01:05:49.521095 21763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:49.522274 21763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:49.523627 21763 hybrid_clock.cc:648] HybridClock initialized: now 1790643949523533 us; error 75 us; skew 500 ppm
I20260929 01:05:49.524107 21763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:49 dist-test-slave-k0b7 krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:49.527623 21763 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:05:49.529425 21763 webserver.cc:460] Webserver started at http://127.3.21.195:46191/ using document root <none> and password file <none>
I20260929 01:05:49.530179 21763 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:49.530334 21763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:49.530745 21763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:49.533949 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3"
format_stamp: "Formatted at 2026-09-29 01:05:49 on dist-test-slave-k0b7"
server_key: "7af663bb6f7e0fc4786c94302a98a1e8"
server_key_iv: "615b43a20470169499ff0a0c5fc46cfa"
server_key_version: "encryptionkey@0"
I20260929 01:05:49.534853 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3"
format_stamp: "Formatted at 2026-09-29 01:05:49 on dist-test-slave-k0b7"
server_key: "7af663bb6f7e0fc4786c94302a98a1e8"
server_key_iv: "615b43a20470169499ff0a0c5fc46cfa"
server_key_version: "encryptionkey@0"
I20260929 01:05:49.540777 21763 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:05:49.545231 21778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:49.547070 21763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:05:49.547291 21763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3"
format_stamp: "Formatted at 2026-09-29 01:05:49 on dist-test-slave-k0b7"
server_key: "7af663bb6f7e0fc4786c94302a98a1e8"
server_key_iv: "615b43a20470169499ff0a0c5fc46cfa"
server_key_version: "encryptionkey@0"
I20260929 01:05:49.547528 21763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:49.560151 21763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:49.565560 21763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:49.565925 21763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:49.567000 21763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:49.568480 21763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:49.568593 21763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:49.568753 21763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:49.568827 21763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:49.607122 21763 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:39245
I20260929 01:05:49.607141 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:39245 every 8 connection(s)
I20260929 01:05:49.608738 21763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:49.617529 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21763
I20260929 01:05:49.617734 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:05:49.618194 3159 external_mini_cluster.cc:1428] Setting key 50dc4991455425ee5246be1a00b28bc2
Sep 29 01:05:49 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:49.632038 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.611675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:57163 (local address 127.3.21.254:46427)
0929 01:05:49.612133 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.612139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.613162 (+ 1023us) server_negotiation.cc:407] Connection header received
0929 01:05:49.615172 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.615178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.615253 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.615374 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:49.617255 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.618385 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.620228 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.620872 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.621042 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.625346 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:49.625353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:49.625356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:49.625395 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:49.628342 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.629317 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.629322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.629324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.629471 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.629910 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.629913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.629915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.630336 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:49.630464 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.631531 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.631723 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260929 01:05:49.633306 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.612220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:46427 (local address 127.3.21.195:57163)
0929 01:05:49.612950 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.612995 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.614849 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.615704 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.615713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.616378 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:49.617002 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.617020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.618604 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.618609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.620014 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.620031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.621233 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.621239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.621676 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.622472 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:49.622594 (+ 122us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:49.625048 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:49.628534 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.628542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.628558 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.629149 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.629602 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.629605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.629608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.629804 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.630558 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:49.630570 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:49.631061 (+ 491us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.631356 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.631809 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":299,"threads_started":1}
I20260929 01:05:49.635241 21891 heartbeater.cc:347] Connected to a master server at 127.3.21.254:46427
I20260929 01:05:49.635555 21891 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:49.636293 21891 heartbeater.cc:511] Master 127.3.21.254:46427 requested a full tablet report, sending...
I20260929 01:05:49.638065 21434 ts_manager.cc:194] Registered new tserver with Master: 9fc7839d6a4b4a779a1ce6e1eb988be3 (127.3.21.195:39245)
I20260929 01:05:49.639106 21434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:57163
I20260929 01:05:49.648231 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:49.667186 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.652928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.3.21.254:46427)
0929 01:05:49.653204 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.653212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.653537 (+ 325us) server_negotiation.cc:407] Connection header received
0929 01:05:49.653766 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.653772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.653849 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.653958 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:49.655570 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.656323 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.657840 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.658265 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.658420 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.661261 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:49.661275 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:49.661279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:49.661315 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:49.664045 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.664917 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.664927 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.664929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.665079 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.665564 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.665571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.665579 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.665897 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:49.666214 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.666626 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.666835 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:05:49.671599 21434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:49.679096 21434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:49.706034 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.696628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.3.21.195:39245)
0929 01:05:49.697404 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.697412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.697444 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:05:49.697547 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.697555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.697810 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.698672 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.698963 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.700470 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.703385 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.704682 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.704836 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.705098 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.705502 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.705668 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":440,"threads_started":1}
I20260929 01:05:49.706130 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.695994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:39245 (local address 127.0.0.1:39520)
0929 01:05:49.696714 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.696734 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.696850 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.698017 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.698022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.698049 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.698740 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.698756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.700626 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.700634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.703244 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.703258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.704886 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.704892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.705199 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.705268 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.705473 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.705631 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":93,"thread_start_us":909,"threads_started":1}
I20260929 01:05:49.706593 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.693944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:34909 (local address 127.0.0.1:57438)
0929 01:05:49.695905 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.695923 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.696151 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.697592 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.697596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.697621 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.698128 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.698140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.700257 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.700263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.702961 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.702981 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.705413 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.705419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.705588 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.705724 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.705882 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.706021 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1821,"mutex_wait_us":481,"thread_start_us":1379,"threads_started":1}
I20260929 01:05:49.706871 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.693945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.3.21.194:34909)
0929 01:05:49.694633 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.694644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.696031 (+ 1387us) server_negotiation.cc:407] Connection header received
0929 01:05:49.696368 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.696380 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.696658 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.696836 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.698397 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.700107 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.703381 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.704850 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.704967 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.705184 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.706254 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.706438 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":727,"threads_started":1}
I20260929 01:05:49.707140 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.693944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36371 (local address 127.0.0.1:57454)
0929 01:05:49.695308 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.695363 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.695609 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.697175 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.697183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.697218 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.697749 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.697765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.699916 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.699924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.702962 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.702981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.705082 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.705088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.705261 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.705295 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.705390 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.705750 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":50,"thread_start_us":813,"threads_started":1}
I20260929 01:05:49.708287 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.694450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.3.21.193:36371)
0929 01:05:49.696384 (+ 1934us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.696396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.696428 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:05:49.696530 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.696537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.696870 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.697988 (+ 1118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.698449 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.699764 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.703190 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.704835 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.707413 (+ 2578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.707669 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.707760 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.707944 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1747,"thread_start_us":1553,"threads_started":1}
I20260929 01:05:49.710969 21692 tablet_service.cc:1467] Processing CreateTablet for tablet 32b46b30f8c74996bdf08e486c89af40 (DEFAULT_TABLE table=test-table [id=d7750da98b6443bc96abd007d8e119a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:49.711257 21826 tablet_service.cc:1467] Processing CreateTablet for tablet 32b46b30f8c74996bdf08e486c89af40 (DEFAULT_TABLE table=test-table [id=d7750da98b6443bc96abd007d8e119a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:49.711663 21558 tablet_service.cc:1467] Processing CreateTablet for tablet 32b46b30f8c74996bdf08e486c89af40 (DEFAULT_TABLE table=test-table [id=d7750da98b6443bc96abd007d8e119a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:49.712776 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b46b30f8c74996bdf08e486c89af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:49.712867 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b46b30f8c74996bdf08e486c89af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:49.713372 21558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b46b30f8c74996bdf08e486c89af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:49.725405 21910 tablet_bootstrap.cc:492] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Bootstrap starting.
I20260929 01:05:49.728711 21912 tablet_bootstrap.cc:492] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Bootstrap starting.
I20260929 01:05:49.730477 21910 tablet_bootstrap.cc:654] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:49.730561 21911 tablet_bootstrap.cc:492] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Bootstrap starting.
I20260929 01:05:49.732023 21910 log.cc:824] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:49.732628 21912 tablet_bootstrap.cc:654] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:49.733834 21912 log.cc:824] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:49.733903 21911 tablet_bootstrap.cc:654] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:49.735189 21911 log.cc:824] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:49.735738 21910 tablet_bootstrap.cc:492] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: No bootstrap required, opened a new log
I20260929 01:05:49.736111 21910 ts_tablet_manager.cc:1397] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260929 01:05:49.737181 21912 tablet_bootstrap.cc:492] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: No bootstrap required, opened a new log
I20260929 01:05:49.737501 21912 ts_tablet_manager.cc:1397] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 01:05:49.748191 21910 raft_consensus.cc:348] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.748732 21910 raft_consensus.cc:374] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:49.748452 21912 raft_consensus.cc:348] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.748871 21910 raft_consensus.cc:729] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd2bef33a174a8ab01015321c8bd2c2, State: Initialized, Role: FOLLOWER
I20260929 01:05:49.748950 21912 raft_consensus.cc:374] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:49.749055 21912 raft_consensus.cc:729] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac23f76ec91043a9a3c6fcc902a35056, State: Initialized, Role: FOLLOWER
I20260929 01:05:49.749595 21910 consensus_queue.cc:260] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.749748 21912 consensus_queue.cc:260] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.749993 21911 tablet_bootstrap.cc:492] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: No bootstrap required, opened a new log
I20260929 01:05:49.750355 21911 ts_tablet_manager.cc:1397] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20260929 01:05:49.767407 21757 heartbeater.cc:503] Master 127.3.21.254:46427 was elected leader, sending a full tablet report...
I20260929 01:05:49.768169 21910 ts_tablet_manager.cc:1428] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Time spent starting tablet: real 0.032s user 0.017s sys 0.013s
I20260929 01:05:49.768028 21911 raft_consensus.cc:348] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.768559 21911 raft_consensus.cc:374] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:49.768661 21912 ts_tablet_manager.cc:1428] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Time spent starting tablet: real 0.031s user 0.008s sys 0.022s
I20260929 01:05:49.768682 21911 raft_consensus.cc:729] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc7839d6a4b4a779a1ce6e1eb988be3, State: Initialized, Role: FOLLOWER
I20260929 01:05:49.769361 21911 consensus_queue.cc:260] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.776701 21891 heartbeater.cc:503] Master 127.3.21.254:46427 was elected leader, sending a full tablet report...
I20260929 01:05:49.777534 21911 ts_tablet_manager.cc:1428] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Time spent starting tablet: real 0.027s user 0.021s sys 0.003s
I20260929 01:05:49.790087 21917 raft_consensus.cc:482] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:49.790592 21917 raft_consensus.cc:504] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.792527 21917 leader_election.cc:288] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abd2bef33a174a8ab01015321c8bd2c2 (127.3.21.194:34909), 9fc7839d6a4b4a779a1ce6e1eb988be3 (127.3.21.195:39245)
I20260929 01:05:49.794194 21916 raft_consensus.cc:482] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:49.794639 21916 raft_consensus.cc:504] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.796905 21916 leader_election.cc:288] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac23f76ec91043a9a3c6fcc902a35056 (127.3.21.193:36371), 9fc7839d6a4b4a779a1ce6e1eb988be3 (127.3.21.195:39245)
I20260929 01:05:49.803861 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.793051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:47521 (local address 127.3.21.194:34909)
0929 01:05:49.793390 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.793401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.794154 (+ 753us) server_negotiation.cc:407] Connection header received
0929 01:05:49.794931 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.794939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.795032 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.795117 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.795994 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.797219 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.800302 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.802047 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.802222 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.802335 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.803184 (+ 849us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.803455 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:05:49.804965 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.792987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:34909 (local address 127.3.21.193:47521)
0929 01:05:49.793868 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.793893 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.794337 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.795346 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.795352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.795385 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.795812 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.795827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.797412 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.797420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.800073 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.800094 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.802647 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.802657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.802930 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.802987 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.804130 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.804369 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":4,"thread_start_us":515,"threads_started":1}
I20260929 01:05:49.806041 21712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b46b30f8c74996bdf08e486c89af40" candidate_uuid: "ac23f76ec91043a9a3c6fcc902a35056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd2bef33a174a8ab01015321c8bd2c2" is_pre_election: true
I20260929 01:05:49.806690 21712 raft_consensus.cc:2454] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac23f76ec91043a9a3c6fcc902a35056 in term 0.
I20260929 01:05:49.807297 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.794339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:52973 (local address 127.3.21.195:39245)
0929 01:05:49.794635 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.794644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.795152 (+ 508us) server_negotiation.cc:407] Connection header received
0929 01:05:49.795448 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.795455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.795546 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.795621 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.796604 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.797852 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.802264 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.805879 (+ 3615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.806045 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.806130 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.806863 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.807009 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 01:05:49.807595 21514 leader_election.cc:302] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abd2bef33a174a8ab01015321c8bd2c2, ac23f76ec91043a9a3c6fcc902a35056; no voters:
I20260929 01:05:49.808103 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.794036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:39245 (local address 127.3.21.193:52973)
0929 01:05:49.795037 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.795057 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.795266 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.795839 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.795843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.795872 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.796346 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.796362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.798332 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.798341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.801940 (+ 3599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.801968 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.806362 (+ 4394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.806371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.806600 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.806634 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.807473 (+ 839us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.807656 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":34,"thread_start_us":621,"threads_started":1}
I20260929 01:05:49.808147 21917 raft_consensus.cc:2781] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:49.808291 21917 raft_consensus.cc:482] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:49.808413 21917 raft_consensus.cc:3037] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:49.809298 21846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b46b30f8c74996bdf08e486c89af40" candidate_uuid: "ac23f76ec91043a9a3c6fcc902a35056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" is_pre_election: true
I20260929 01:05:49.809911 21846 raft_consensus.cc:2454] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac23f76ec91043a9a3c6fcc902a35056 in term 0.
I20260929 01:05:49.812042 21917 raft_consensus.cc:504] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.812772 21917 leader_election.cc:288] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [CANDIDATE]: Term 1 election: Requested vote from peers abd2bef33a174a8ab01015321c8bd2c2 (127.3.21.194:34909), 9fc7839d6a4b4a779a1ce6e1eb988be3 (127.3.21.195:39245)
I20260929 01:05:49.813423 21846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b46b30f8c74996bdf08e486c89af40" candidate_uuid: "ac23f76ec91043a9a3c6fcc902a35056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3"
I20260929 01:05:49.813437 21712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b46b30f8c74996bdf08e486c89af40" candidate_uuid: "ac23f76ec91043a9a3c6fcc902a35056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd2bef33a174a8ab01015321c8bd2c2"
I20260929 01:05:49.813657 21846 raft_consensus.cc:3037] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:49.813732 21712 raft_consensus.cc:3037] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:05:49.815713 21758 tablet.cc:2369] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:49.817258 21846 raft_consensus.cc:2454] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac23f76ec91043a9a3c6fcc902a35056 in term 1.
I20260929 01:05:49.817776 21514 leader_election.cc:302] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fc7839d6a4b4a779a1ce6e1eb988be3, ac23f76ec91043a9a3c6fcc902a35056; no voters:
I20260929 01:05:49.817833 21712 raft_consensus.cc:2454] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac23f76ec91043a9a3c6fcc902a35056 in term 1.
I20260929 01:05:49.818243 21917 raft_consensus.cc:2781] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:49.818706 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.797741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:39245 (local address 127.3.21.194:45689)
0929 01:05:49.809780 (+ 12039us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.809799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.809941 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.810695 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.810700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.810735 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.811310 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.811328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.812933 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.812939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.815475 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.815491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.817710 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.817718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.817912 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.817945 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.818032 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.818186 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11910,"thread_start_us":12025,"threads_started":1}
I20260929 01:05:49.819068 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.798339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:45689 (local address 127.3.21.195:39245)
0929 01:05:49.806040 (+ 7701us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.806050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.810029 (+ 3979us) server_negotiation.cc:407] Connection header received
0929 01:05:49.810158 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.810163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.810257 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.810361 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.811567 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.812753 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.815703 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.817184 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.817350 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.817427 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.818570 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.818731 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7483,"thread_start_us":7181,"threads_started":1}
I20260929 01:05:49.819586 21846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b46b30f8c74996bdf08e486c89af40" candidate_uuid: "abd2bef33a174a8ab01015321c8bd2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" is_pre_election: true
I20260929 01:05:49.819809 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.798166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:57943 (local address 127.3.21.193:36371)
0929 01:05:49.798714 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.798721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.809312 (+ 10591us) server_negotiation.cc:407] Connection header received
0929 01:05:49.810268 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.810273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.810351 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.810415 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:49.811407 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.812330 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.816724 (+ 4394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.818102 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.818235 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.818302 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.819417 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.819563 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260929 01:05:49.819911 21846 raft_consensus.cc:2379] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate abd2bef33a174a8ab01015321c8bd2c2 in current term 1: Already voted for candidate ac23f76ec91043a9a3c6fcc902a35056 in this term.
I20260929 01:05:49.820014 21917 raft_consensus.cc:686] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 1 LEADER]: Becoming Leader. State: Replica: ac23f76ec91043a9a3c6fcc902a35056, State: Running, Role: LEADER
I20260929 01:05:49.820211 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.797426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36371 (local address 127.3.21.194:57943)
0929 01:05:49.809213 (+ 11787us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.809233 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.810117 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.810642 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.810647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.810677 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:49.811186 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.811199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.812469 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.812476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.815242 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.815258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.818674 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.818680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.819067 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.819109 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.819376 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.819523 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11644,"mutex_wait_us":118,"thread_start_us":12053,"threads_started":1}
I20260929 01:05:49.820611 21917 consensus_queue.cc:237] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } }
I20260929 01:05:49.821115 21578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b46b30f8c74996bdf08e486c89af40" candidate_uuid: "abd2bef33a174a8ab01015321c8bd2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac23f76ec91043a9a3c6fcc902a35056" is_pre_election: true
I20260929 01:05:49.822141 21645 leader_election.cc:302] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abd2bef33a174a8ab01015321c8bd2c2; no voters: 9fc7839d6a4b4a779a1ce6e1eb988be3, ac23f76ec91043a9a3c6fcc902a35056
I20260929 01:05:49.822669 21916 raft_consensus.cc:2726] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:05:49.826249 21433 catalog_manager.cc:5495] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac23f76ec91043a9a3c6fcc902a35056 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "ac23f76ec91043a9a3c6fcc902a35056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac23f76ec91043a9a3c6fcc902a35056" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 } health_report { overall_health: UNKNOWN } } }
W20260929 01:05:49.860591 21892 tablet.cc:2369] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:49.871825 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.864346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.3.21.193:36371)
0929 01:05:49.864546 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.864551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.864682 (+ 131us) server_negotiation.cc:407] Connection header received
0929 01:05:49.865050 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.865056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.865162 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.865304 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:49.866059 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.866969 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.868685 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.869275 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.869390 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.870243 (+ 853us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:49.870995 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:49.871144 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.871367 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.871485 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:05:49.881433 21712 raft_consensus.cc:1261] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 1 FOLLOWER]: Refusing update from remote peer ac23f76ec91043a9a3c6fcc902a35056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:49.881454 21846 raft_consensus.cc:1261] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 1 FOLLOWER]: Refusing update from remote peer ac23f76ec91043a9a3c6fcc902a35056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:49.882817 21924 consensus_queue.cc:1035] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fc7839d6a4b4a779a1ce6e1eb988be3" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:49.883248 21917 consensus_queue.cc:1035] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd2bef33a174a8ab01015321c8bd2c2" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:49.900497 21929 mvcc.cc:205] Tried to move back new op lower bound from 7334477618701783040 to 7334477618469609472. Current Snapshot: MvccSnapshot[applied={T|T < 7334477618701783040}]
I20260929 01:05:49.916965 21434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:49.917332 21434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:49.923693 21434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9952bcfceb2940b785b4bc7987187235: Sending DeleteTablet for 3 replicas of tablet 32b46b30f8c74996bdf08e486c89af40
I20260929 01:05:49.925405 21692 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b46b30f8c74996bdf08e486c89af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:49 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:57438
I20260929 01:05:49.926469 21936 tablet_replica.cc:330] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: stopping tablet replica
I20260929 01:05:49.927212 21936 raft_consensus.cc:2229] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:49.927403 21558 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b46b30f8c74996bdf08e486c89af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:49 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:57454
I20260929 01:05:49.927799 21936 raft_consensus.cc:2258] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:49.930665 21936 ts_tablet_manager.cc:1905] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:49.931684 21826 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b46b30f8c74996bdf08e486c89af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:49 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:39520
I20260929 01:05:49.933837 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21495
I20260929 01:05:49.942318 21938 tablet_replica.cc:330] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: stopping tablet replica
I20260929 01:05:49.942855 21937 tablet_replica.cc:330] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: stopping tablet replica
I20260929 01:05:49.943329 21937 raft_consensus.cc:2229] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:49.943325 21938 raft_consensus.cc:2229] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:49.944119 21937 raft_consensus.cc:2258] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:49.944156 21938 raft_consensus.cc:2258] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:49.946542 21937 ts_tablet_manager.cc:1905] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:49.946633 21938 ts_tablet_manager.cc:1905] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:49.954610 21937 ts_tablet_manager.cc:1918] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:49.954818 21937 log.cc:1196] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/32b46b30f8c74996bdf08e486c89af40
I20260929 01:05:49.955005 21938 ts_tablet_manager.cc:1918] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:49.955309 21938 log.cc:1196] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/32b46b30f8c74996bdf08e486c89af40
I20260929 01:05:49.955401 21937 ts_tablet_manager.cc:1939] T 32b46b30f8c74996bdf08e486c89af40 P ac23f76ec91043a9a3c6fcc902a35056: Deleting consensus metadata
I20260929 01:05:49.955955 21938 ts_tablet_manager.cc:1939] T 32b46b30f8c74996bdf08e486c89af40 P 9fc7839d6a4b4a779a1ce6e1eb988be3: Deleting consensus metadata
I20260929 01:05:49.958182 21418 catalog_manager.cc:4841] TS ac23f76ec91043a9a3c6fcc902a35056 (127.3.21.193:36371): tablet 32b46b30f8c74996bdf08e486c89af40 (table test-table [id=d7750da98b6443bc96abd007d8e119a2]) successfully deleted
I20260929 01:05:49.957989 21421 catalog_manager.cc:4841] TS 9fc7839d6a4b4a779a1ce6e1eb988be3 (127.3.21.195:39245): tablet 32b46b30f8c74996bdf08e486c89af40 (table test-table [id=d7750da98b6443bc96abd007d8e119a2]) successfully deleted
I20260929 01:05:49.961598 21936 ts_tablet_manager.cc:1918] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:49.961846 21936 log.cc:1196] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/32b46b30f8c74996bdf08e486c89af40
I20260929 01:05:49.962606 21936 ts_tablet_manager.cc:1939] T 32b46b30f8c74996bdf08e486c89af40 P abd2bef33a174a8ab01015321c8bd2c2: Deleting consensus metadata
I20260929 01:05:49.964727 21421 catalog_manager.cc:4841] TS abd2bef33a174a8ab01015321c8bd2c2 (127.3.21.194:34909): tablet 32b46b30f8c74996bdf08e486c89af40 (table test-table [id=d7750da98b6443bc96abd007d8e119a2]) successfully deleted
Sep 29 01:05:49 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:05:49.979130 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.936411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.3.21.193:36371)
0929 01:05:49.936660 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.936670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.953487 (+ 16817us) server_negotiation.cc:407] Connection header received
0929 01:05:49.953999 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.954005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.954136 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.954214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:49.956257 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.957369 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.962683 (+ 5314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.963377 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.963511 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.970221 (+ 6710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:49.970244 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:49.970268 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:49.970323 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:49.975335 (+ 5012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.976361 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.976371 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.976375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.976568 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.977105 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.977110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.977114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.977648 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:49.978023 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.978354 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.978739 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:05:49.979935 21618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:50.096525 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21495
I20260929 01:05:50.116961 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21629
Sep 29 01:05:50 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:05:50.136049 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:50.117820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.3.21.194:34909)
0929 01:05:50.118055 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:50.118062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:50.118284 (+ 222us) server_negotiation.cc:407] Connection header received
0929 01:05:50.118615 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:50.118623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:50.118723 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:50.118839 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:50.120200 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:50.120983 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:50.122327 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:50.122902 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:50.123050 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:50.128091 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:50.128110 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:50.128127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:50.128174 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:50.131726 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:50.132725 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:50.132734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:50.132736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:50.132905 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:50.133600 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:50.133606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:50.133609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:50.134234 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:50.134612 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:50.135243 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 01:05:50.135582 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 01:05:50.137014 21752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:50.233109 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21629
I20260929 01:05:50.251749 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21763
Sep 29 01:05:50 dist-test-slave-k0b7 krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:05:50.271742 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:50.252763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.3.21.195:39245)
0929 01:05:50.253088 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:50.253093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:50.253270 (+ 177us) server_negotiation.cc:407] Connection header received
0929 01:05:50.253497 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:50.253504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:50.253619 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:50.253735 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:50.255557 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:50.256375 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:50.258295 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:50.258919 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:50.259062 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:50.264666 (+ 5604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:50.264685 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:50.264703 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:50.264765 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:50.268109 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:50.269135 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:50.269144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:50.269147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:50.269297 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:50.269771 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:50.269775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:50.269777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:50.270320 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:50.270668 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:50.271063 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 01:05:50.271309 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:05:50.272605 21886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:50.366811 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21763
I20260929 01:05:50.384348 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21402
I20260929 01:05:50.385664 21464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:50.504160 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21402
2026-09-29T01:05:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4711 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T01:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:50Z Disabled control of system clock
I20260929 01:05:50.586596 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34147
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:05:50.958178 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:50.958528 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:50.958647 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:50.966246 21947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:50.966400 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:50.966481 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:50.966622 21947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:50.966691 21947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:50.977376 21947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:50.978780 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:50.980693 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:50.993435 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:50.993579 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:50.994004 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:50.994899 21947 server_base.cc:1034] running on GCE node
I20260929 01:05:50.995759 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:50.997386 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:50.998687 21947 hybrid_clock.cc:648] HybridClock initialized: now 1790643950998630 us; error 53 us; skew 500 ppm
I20260929 01:05:50.999104 21947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:51.001742 21947 webserver.cc:460] Webserver started at http://127.3.21.254:40261/ using document root <none> and password file <none>
I20260929 01:05:51.002732 21947 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:51.002913 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:51.003479 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:51.007185 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "22411af8676e44279d8a8fa3982e7d53"
format_stamp: "Formatted at 2026-09-29 01:05:50 on dist-test-slave-k0b7"
server_key: "9356f84ef7ae524f8e48d6f4d329fc90"
server_key_iv: "841bfffad534ca703e66488cb66fbdcb"
server_key_version: "encryptionkey@0"
I20260929 01:05:51.008167 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "22411af8676e44279d8a8fa3982e7d53"
format_stamp: "Formatted at 2026-09-29 01:05:50 on dist-test-slave-k0b7"
server_key: "9356f84ef7ae524f8e48d6f4d329fc90"
server_key_iv: "841bfffad534ca703e66488cb66fbdcb"
server_key_version: "encryptionkey@0"
I20260929 01:05:51.014691 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 01:05:51.018879 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:51.020812 21947 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:05:51.021052 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "22411af8676e44279d8a8fa3982e7d53"
format_stamp: "Formatted at 2026-09-29 01:05:50 on dist-test-slave-k0b7"
server_key: "9356f84ef7ae524f8e48d6f4d329fc90"
server_key_iv: "841bfffad534ca703e66488cb66fbdcb"
server_key_version: "encryptionkey@0"
I20260929 01:05:51.021311 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:51.040498 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:51.041543 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:51.041864 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:51.063441 21947 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34147
I20260929 01:05:51.063488 22012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34147 every 8 connection(s)
I20260929 01:05:51.064838 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:51.068776 22013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:51.074883 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 21947
I20260929 01:05:51.075081 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:05:51.075665 3159 external_mini_cluster.cc:1428] Setting key b97cd264dd847865a462fcdef903d6ba
I20260929 01:05:51.080781 22013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53: Bootstrap starting.
I20260929 01:05:51.084131 22013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:51.085160 22013 log.cc:824] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:51.088941 22013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53: No bootstrap required, opened a new log
I20260929 01:05:51.091151 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:51.077822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.3.21.254:34147)
0929 01:05:51.078956 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:51.078968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:51.079020 (+ 52us) server_negotiation.cc:407] Connection header received
0929 01:05:51.080885 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:51.080919 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:51.081428 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:51.081841 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:51.082487 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:51.083881 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:51.085401 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:51.086443 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:51.086678 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:51.088064 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:51.088074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:51.088093 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:51.088847 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:51.088961 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:51.088992 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:51.089108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:51.089456 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 01:05:51.089829 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":332,"threads_started":1}
I20260929 01:05:51.098070 22013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22411af8676e44279d8a8fa3982e7d53" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34147 } }
I20260929 01:05:51.098582 22013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:51.098673 22013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22411af8676e44279d8a8fa3982e7d53, State: Initialized, Role: FOLLOWER
I20260929 01:05:51.099259 22013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22411af8676e44279d8a8fa3982e7d53" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34147 } }
I20260929 01:05:51.099486 22013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:51.099579 22013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:51.099704 22013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:51.102425 22013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22411af8676e44279d8a8fa3982e7d53" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34147 } }
I20260929 01:05:51.102989 22013 leader_election.cc:302] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22411af8676e44279d8a8fa3982e7d53; no voters:
I20260929 01:05:51.103588 22013 leader_election.cc:288] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:51.103772 22018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:51.104607 22018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [term 1 LEADER]: Becoming Leader. State: Replica: 22411af8676e44279d8a8fa3982e7d53, State: Running, Role: LEADER
I20260929 01:05:51.105163 22018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22411af8676e44279d8a8fa3982e7d53" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34147 } }
I20260929 01:05:51.105818 22013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:51.109086 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22411af8676e44279d8a8fa3982e7d53. Latest consensus state: current_term: 1 leader_uuid: "22411af8676e44279d8a8fa3982e7d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22411af8676e44279d8a8fa3982e7d53" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34147 } } }
I20260929 01:05:51.109546 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:51.110909 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22411af8676e44279d8a8fa3982e7d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22411af8676e44279d8a8fa3982e7d53" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34147 } } }
I20260929 01:05:51.111311 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:51.114861 22027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:51.121153 22027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:51.131328 22027 catalog_manager.cc:1347] Generated new cluster ID: 92758eabb2064abf8b5a3a51024abf2d
I20260929 01:05:51.131492 22027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:51.150354 22027 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:51.151875 22027 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:51.169166 22027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53: Generated new TSK 0
I20260929 01:05:51.170452 22027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:05:51.183012 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34147
--builtin_ntp_servers=127.3.21.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:05:51.566236 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:51.566687 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:51.566871 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:51.575299 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:51.575459 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:51.575742 22037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:51.585614 22037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:51.587095 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:51.588995 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:51.603663 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:51.603565 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:51.603600 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:51.605197 22037 server_base.cc:1034] running on GCE node
I20260929 01:05:51.606051 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:51.607493 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:51.608834 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790643951608764 us; error 66 us; skew 500 ppm
I20260929 01:05:51.609428 22037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:51.612563 22037 webserver.cc:460] Webserver started at http://127.3.21.193:43859/ using document root <none> and password file <none>
I20260929 01:05:51.613366 22037 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:51.613525 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:51.613883 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:51.617127 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "77c9543fee6f41e2928f7cf1efd83211"
format_stamp: "Formatted at 2026-09-29 01:05:51 on dist-test-slave-k0b7"
server_key: "e549c2c5591255237e3e418c422e01ff"
server_key_iv: "6e5d918ffa6396bd3acbbb4829f5d490"
server_key_version: "encryptionkey@0"
I20260929 01:05:51.618006 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "77c9543fee6f41e2928f7cf1efd83211"
format_stamp: "Formatted at 2026-09-29 01:05:51 on dist-test-slave-k0b7"
server_key: "e549c2c5591255237e3e418c422e01ff"
server_key_iv: "6e5d918ffa6396bd3acbbb4829f5d490"
server_key_version: "encryptionkey@0"
I20260929 01:05:51.624719 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260929 01:05:51.629283 22051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:51.631106 22037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:51.631276 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "77c9543fee6f41e2928f7cf1efd83211"
format_stamp: "Formatted at 2026-09-29 01:05:51 on dist-test-slave-k0b7"
server_key: "e549c2c5591255237e3e418c422e01ff"
server_key_iv: "6e5d918ffa6396bd3acbbb4829f5d490"
server_key_version: "encryptionkey@0"
I20260929 01:05:51.631512 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:51.652066 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:51.652978 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:51.653270 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:51.654740 22037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:51.656654 22037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:51.656785 22037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:51.656908 22037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:51.656999 22037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:51.697160 22037 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:45899
I20260929 01:05:51.697222 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:45899 every 8 connection(s)
I20260929 01:05:51.698747 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:51.704996 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22037
I20260929 01:05:51.705207 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:51.705874 3159 external_mini_cluster.cc:1428] Setting key cf63e8ef73387f0954146ba668042bd5
I20260929 01:05:51.709666 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34147
--builtin_ntp_servers=127.3.21.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:05:51.717458 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:51.702124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:44535 (local address 127.3.21.254:34147)
0929 01:05:51.702701 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:51.702708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:51.704100 (+ 1392us) server_negotiation.cc:407] Connection header received
0929 01:05:51.707080 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:51.707278 (+ 198us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:51.707391 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:51.707523 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:51.709345 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:51.710322 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:51.713539 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:51.714105 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:51.714323 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:51.716392 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:51.716399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:51.716404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:51.716426 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:51.716473 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:51.716486 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:51.716570 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:51.717077 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 01:05:51.717185 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":253,"threads_started":1}
I20260929 01:05:51.718894 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:51.702610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34147 (local address 127.3.21.193:44535)
0929 01:05:51.703903 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:51.703963 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:51.706631 (+ 2668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:51.707820 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:51.707830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:51.708190 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:51.709122 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:51.709151 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:51.710561 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:51.710570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:51.713287 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:51.713312 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:51.714458 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:51.714468 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:51.714974 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:51.715938 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:51.715944 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:51.716004 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:51.716136 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:51.716605 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:51.716620 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:51.716632 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:05:51.716837 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 01:05:51.717347 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":537,"threads_started":1}
I20260929 01:05:51.721627 22164 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34147
I20260929 01:05:51.721942 22164 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:51.722885 22164 heartbeater.cc:511] Master 127.3.21.254:34147 requested a full tablet report, sending...
I20260929 01:05:51.725956 21978 ts_manager.cc:194] Registered new tserver with Master: 77c9543fee6f41e2928f7cf1efd83211 (127.3.21.193:45899)
I20260929 01:05:51.728173 21978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.193:44535
W20260929 01:05:52.103395 22168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:52.103785 22168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:52.103950 22168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:52.111994 22168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:52.112169 22168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:52.112365 22168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:05:52.123028 22168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:52.124424 22168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:52.126636 22168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:52.142182 22173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:52.142100 22176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:52.142261 22174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:52.144948 22168 server_base.cc:1034] running on GCE node
I20260929 01:05:52.145972 22168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:52.147396 22168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:52.148872 22168 hybrid_clock.cc:648] HybridClock initialized: now 1790643952148793 us; error 82 us; skew 500 ppm
I20260929 01:05:52.149272 22168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:52.152344 22168 webserver.cc:460] Webserver started at http://127.3.21.194:42141/ using document root <none> and password file <none>
I20260929 01:05:52.153146 22168 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:52.153287 22168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:52.153626 22168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:52.156837 22168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "4e44c9db111b4433b5e2350a2553e17a"
format_stamp: "Formatted at 2026-09-29 01:05:52 on dist-test-slave-k0b7"
server_key: "12e1af9db73da764d43fd40b6aa51c6a"
server_key_iv: "0f74178e1d50ff6a2de11c22ac7e69d9"
server_key_version: "encryptionkey@0"
I20260929 01:05:52.158047 22168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "4e44c9db111b4433b5e2350a2553e17a"
format_stamp: "Formatted at 2026-09-29 01:05:52 on dist-test-slave-k0b7"
server_key: "12e1af9db73da764d43fd40b6aa51c6a"
server_key_iv: "0f74178e1d50ff6a2de11c22ac7e69d9"
server_key_version: "encryptionkey@0"
I20260929 01:05:52.164201 22168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260929 01:05:52.168615 22182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:52.170459 22168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 01:05:52.170747 22168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "4e44c9db111b4433b5e2350a2553e17a"
format_stamp: "Formatted at 2026-09-29 01:05:52 on dist-test-slave-k0b7"
server_key: "12e1af9db73da764d43fd40b6aa51c6a"
server_key_iv: "0f74178e1d50ff6a2de11c22ac7e69d9"
server_key_version: "encryptionkey@0"
I20260929 01:05:52.171006 22168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:52.185078 22168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:52.186062 22168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:52.186389 22168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:52.187543 22168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:52.189116 22168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:52.189246 22168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:52.189388 22168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:52.189478 22168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:52.227105 22168 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:38327
I20260929 01:05:52.227141 22294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:38327 every 8 connection(s)
I20260929 01:05:52.228578 22168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:52.238694 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22168
I20260929 01:05:52.238866 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:05:52.239444 3159 external_mini_cluster.cc:1428] Setting key 38cb85b79d178d4efe15fe21408f3640
I20260929 01:05:52.243126 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34147
--builtin_ntp_servers=127.3.21.212:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:05:52.244545 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.231657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:44123 (local address 127.3.21.254:34147)
0929 01:05:52.232119 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.232124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.233463 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 01:05:52.235522 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.235530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.235638 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.235778 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:52.237327 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.238218 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.240446 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.241112 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.241276 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.243400 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:52.243408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:52.243413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:52.243443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:52.243511 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:52.243529 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:52.243620 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.244139 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.244244 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260929 01:05:52.246023 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.232310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34147 (local address 127.3.21.194:44123)
0929 01:05:52.233288 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.233325 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.235170 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.236186 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.236195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.236296 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:52.237087 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.237108 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.238438 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.238444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.240267 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.240286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.241483 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.241496 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.241801 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.242869 (+ 1068us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:52.242875 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:52.242943 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:52.243063 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:52.243651 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:52.243703 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:52.243716 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.243930 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.244456 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":324,"threads_started":1}
I20260929 01:05:52.248006 22295 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34147
I20260929 01:05:52.248414 22295 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:52.249346 22295 heartbeater.cc:511] Master 127.3.21.254:34147 requested a full tablet report, sending...
I20260929 01:05:52.251102 21978 ts_manager.cc:194] Registered new tserver with Master: 4e44c9db111b4433b5e2350a2553e17a (127.3.21.194:38327)
I20260929 01:05:52.252053 21978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.194:44123
W20260929 01:05:52.602738 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:52.603092 22299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:52.603248 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:52.611399 22299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:52.611524 22299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:52.611698 22299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:05:52.621536 22299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:52.622990 22299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:52.624913 22299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:52.640784 22305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:52.640825 22304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:52.640852 22307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:52.641782 22299 server_base.cc:1034] running on GCE node
I20260929 01:05:52.642627 22299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:52.643733 22299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:52.645025 22299 hybrid_clock.cc:648] HybridClock initialized: now 1790643952644953 us; error 65 us; skew 500 ppm
I20260929 01:05:52.645476 22299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:52.648425 22299 webserver.cc:460] Webserver started at http://127.3.21.195:36213/ using document root <none> and password file <none>
I20260929 01:05:52.649255 22299 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:52.649395 22299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:52.649744 22299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:52.653092 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "cee210061f714e0eb880707328129008"
format_stamp: "Formatted at 2026-09-29 01:05:52 on dist-test-slave-k0b7"
server_key: "47a8147b76026b02e7006e863ba3b17d"
server_key_iv: "771e706da22a1d3812249ec774eb8d59"
server_key_version: "encryptionkey@0"
I20260929 01:05:52.653954 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "cee210061f714e0eb880707328129008"
format_stamp: "Formatted at 2026-09-29 01:05:52 on dist-test-slave-k0b7"
server_key: "47a8147b76026b02e7006e863ba3b17d"
server_key_iv: "771e706da22a1d3812249ec774eb8d59"
server_key_version: "encryptionkey@0"
I20260929 01:05:52.659649 22299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 01:05:52.663848 22313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:52.665771 22299 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:05:52.666046 22299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "cee210061f714e0eb880707328129008"
format_stamp: "Formatted at 2026-09-29 01:05:52 on dist-test-slave-k0b7"
server_key: "47a8147b76026b02e7006e863ba3b17d"
server_key_iv: "771e706da22a1d3812249ec774eb8d59"
server_key_version: "encryptionkey@0"
I20260929 01:05:52.666358 22299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:52.687489 22299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:52.688494 22299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:52.688886 22299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:52.690207 22299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:52.692168 22299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:52.692322 22299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:52.692497 22299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:52.692601 22299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:52.732141 22299 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:46233
I20260929 01:05:52.732234 22425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:46233 every 8 connection(s)
I20260929 01:05:52.733100 22164 heartbeater.cc:503] Master 127.3.21.254:34147 was elected leader, sending a full tablet report...
I20260929 01:05:52.733685 22299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:52.734366 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22299
I20260929 01:05:52.734565 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:05:52.735323 3159 external_mini_cluster.cc:1428] Setting key 6d823e515c284128cd2a44ac11899b57
I20260929 01:05:52.748137 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.737390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:53671 (local address 127.3.21.254:34147)
0929 01:05:52.737580 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.737588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.739237 (+ 1649us) server_negotiation.cc:407] Connection header received
0929 01:05:52.741125 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.741130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.741222 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.741339 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:52.742445 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.743244 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.744830 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.745308 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.745581 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.747273 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:52.747279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:52.747284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:52.747315 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:52.747351 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:52.747363 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:52.747463 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.747702 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.747810 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:05:52.749261 22428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.738001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34147 (local address 127.3.21.195:53671)
0929 01:05:52.739074 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.739117 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.740837 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.741576 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.741585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.741676 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:52.742281 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.742299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.743413 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.743417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.744680 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.744692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.745664 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.745672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.745935 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.746916 (+ 981us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:52.746919 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:52.746960 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:52.747047 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:52.747460 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:52.747472 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:52.747483 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.747691 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.748145 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":330,"threads_started":1}
I20260929 01:05:52.750890 22426 heartbeater.cc:347] Connected to a master server at 127.3.21.254:34147
I20260929 01:05:52.751245 22426 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:52.752032 22426 heartbeater.cc:511] Master 127.3.21.254:34147 requested a full tablet report, sending...
I20260929 01:05:52.753538 21977 ts_manager.cc:194] Registered new tserver with Master: cee210061f714e0eb880707328129008 (127.3.21.195:46233)
I20260929 01:05:52.754542 21977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.195:53671
I20260929 01:05:52.764859 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:52.784600 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.776695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.3.21.254:34147)
0929 01:05:52.776978 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.776985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.777214 (+ 229us) server_negotiation.cc:407] Connection header received
0929 01:05:52.777530 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.777536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.777637 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.777762 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:52.778592 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.779690 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.781253 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.781732 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.781888 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.783581 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:52.783587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:52.783590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:52.783612 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:52.783659 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:52.783673 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:52.783907 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.784159 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.784235 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:05:52.789904 21977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:52.797811 21977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:52.828171 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.813767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:38327 (local address 127.0.0.1:38712)
0929 01:05:52.815699 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.815745 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.816026 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.817285 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.817297 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.817344 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:52.817951 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.817973 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.820795 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.820804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.824504 (+ 3700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.824534 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.827106 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.827113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.827359 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.827422 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.827547 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.827763 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1679,"thread_start_us":1528,"threads_started":1}
I20260929 01:05:52.829064 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.813948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.3.21.194:38327)
0929 01:05:52.815087 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.815103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.815855 (+ 752us) server_negotiation.cc:407] Connection header received
0929 01:05:52.816312 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.816324 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.816621 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.816819 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:52.818249 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.820495 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.824838 (+ 4343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.826852 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.827912 (+ 1060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.828371 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.828477 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.828743 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":1108,"threads_started":1}
I20260929 01:05:52.833685 22230 tablet_service.cc:1467] Processing CreateTablet for tablet 135d668d938a47369d1260581337d550 (DEFAULT_TABLE table=test-table [id=540c832df0db4190bba26e4934262ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:52.834896 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.817540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:46233 (local address 127.0.0.1:34946)
0929 01:05:52.826024 (+ 8484us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.826050 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.826168 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.827767 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.827772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.827799 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:52.828160 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.828172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.830049 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.830054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.832337 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.832350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.834064 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.834070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.834235 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.834268 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.834356 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.834553 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8363,"thread_start_us":8238,"threads_started":1}
I20260929 01:05:52.835507 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135d668d938a47369d1260581337d550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:52.835883 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.814155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35270)
0929 01:05:52.825273 (+ 11118us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.825295 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.825452 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.826311 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.826319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.826350 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:52.826824 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.826844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.829541 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.829548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.832194 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.832217 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.834391 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.834398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.834763 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.834801 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.834876 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.835397 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10983,"mutex_wait_us":75,"thread_start_us":10896,"threads_started":1}
I20260929 01:05:52.836143 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.816545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.3.21.195:46233)
0929 01:05:52.826850 (+ 10305us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.826858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.826897 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:05:52.826998 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.827006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.827261 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.827458 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:52.828386 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.829882 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.832512 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.833836 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.834032 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.835494 (+ 1462us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.835580 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.835769 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10110,"thread_start_us":9922,"threads_started":1}
I20260929 01:05:52.837165 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.814903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.3.21.193:45899)
0929 01:05:52.825026 (+ 10123us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.825036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.825396 (+ 360us) server_negotiation.cc:407] Connection header received
0929 01:05:52.825651 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.825662 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.826013 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.827103 (+ 1090us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:52.827383 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.829375 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.832457 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.834123 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.836437 (+ 2314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.836722 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.836780 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.836915 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9883,"thread_start_us":9734,"threads_started":1}
I20260929 01:05:52.840721 22361 tablet_service.cc:1467] Processing CreateTablet for tablet 135d668d938a47369d1260581337d550 (DEFAULT_TABLE table=test-table [id=540c832df0db4190bba26e4934262ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:52.841068 22099 tablet_service.cc:1467] Processing CreateTablet for tablet 135d668d938a47369d1260581337d550 (DEFAULT_TABLE table=test-table [id=540c832df0db4190bba26e4934262ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:52.842106 22361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135d668d938a47369d1260581337d550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:52.842657 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135d668d938a47369d1260581337d550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:52.852870 22444 tablet_bootstrap.cc:492] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Bootstrap starting.
I20260929 01:05:52.855213 22445 tablet_bootstrap.cc:492] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Bootstrap starting.
I20260929 01:05:52.856966 22444 tablet_bootstrap.cc:654] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:52.858245 22444 log.cc:824] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:52.859547 22445 tablet_bootstrap.cc:654] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:52.861236 22445 log.cc:824] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:52.863138 22446 tablet_bootstrap.cc:492] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Bootstrap starting.
I20260929 01:05:52.865288 22445 tablet_bootstrap.cc:492] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: No bootstrap required, opened a new log
I20260929 01:05:52.865613 22445 ts_tablet_manager.cc:1397] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 01:05:52.867007 22444 tablet_bootstrap.cc:492] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: No bootstrap required, opened a new log
I20260929 01:05:52.867347 22444 ts_tablet_manager.cc:1397] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260929 01:05:52.868253 22446 tablet_bootstrap.cc:654] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:52.869791 22446 log.cc:824] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:52.873406 22446 tablet_bootstrap.cc:492] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: No bootstrap required, opened a new log
I20260929 01:05:52.873729 22446 ts_tablet_manager.cc:1397] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260929 01:05:52.875887 22445 raft_consensus.cc:348] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.876417 22445 raft_consensus.cc:374] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:52.876533 22445 raft_consensus.cc:729] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee210061f714e0eb880707328129008, State: Initialized, Role: FOLLOWER
I20260929 01:05:52.877249 22445 consensus_queue.cc:260] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.878573 22426 heartbeater.cc:503] Master 127.3.21.254:34147 was elected leader, sending a full tablet report...
I20260929 01:05:52.879377 22445 ts_tablet_manager.cc:1428] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 01:05:52.880970 22444 raft_consensus.cc:348] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.881423 22444 raft_consensus.cc:374] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:52.881572 22444 raft_consensus.cc:729] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e44c9db111b4433b5e2350a2553e17a, State: Initialized, Role: FOLLOWER
I20260929 01:05:52.882253 22444 consensus_queue.cc:260] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.884124 22444 ts_tablet_manager.cc:1428] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260929 01:05:52.884321 22295 heartbeater.cc:503] Master 127.3.21.254:34147 was elected leader, sending a full tablet report...
I20260929 01:05:52.884510 22446 raft_consensus.cc:348] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.884984 22446 raft_consensus.cc:374] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:52.885100 22446 raft_consensus.cc:729] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c9543fee6f41e2928f7cf1efd83211, State: Initialized, Role: FOLLOWER
I20260929 01:05:52.885766 22446 consensus_queue.cc:260] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.887712 22446 ts_tablet_manager.cc:1428] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260929 01:05:52.898599 22451 raft_consensus.cc:482] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:52.898874 22451 raft_consensus.cc:504] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.900764 22451 leader_election.cc:288] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77c9543fee6f41e2928f7cf1efd83211 (127.3.21.193:45899), cee210061f714e0eb880707328129008 (127.3.21.195:46233)
I20260929 01:05:52.909719 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.901120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:46233 (local address 127.3.21.194:41257)
0929 01:05:52.901970 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.901991 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.902141 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.903066 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.903070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.903109 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:52.903637 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.903648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.904937 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.904942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.906789 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.906802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.908828 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.908837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.909011 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.909048 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.909199 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.909360 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":47,"thread_start_us":546,"threads_started":1}
I20260929 01:05:52.909848 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.901353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:41257 (local address 127.3.21.195:46233)
0929 01:05:52.901562 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.901569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.902078 (+ 509us) server_negotiation.cc:407] Connection header received
0929 01:05:52.902566 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.902574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.902856 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.903021 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:52.903861 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.904751 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.907005 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.908176 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.908352 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.908590 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.909255 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.909549 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:05:52.910158 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.901145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:33385 (local address 127.3.21.193:45899)
0929 01:05:52.901459 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.901466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.901922 (+ 456us) server_negotiation.cc:407] Connection header received
0929 01:05:52.902183 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.902191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.902306 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.902387 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:52.903789 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.904565 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.907386 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.908326 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.908521 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.908592 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.909478 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.909643 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 01:05:52.910590 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.900932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.3.21.194:33385)
0929 01:05:52.901790 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.901812 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:52.901975 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:52.902568 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:52.902575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:52.902632 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:52.903614 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.903632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.904716 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.904722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.907053 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:52.907070 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.908901 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.908926 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:52.909220 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.909271 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 01:05:52.910034 (+ 763us) client_negotiation.cc:240] Negotiation successful
0929 01:05:52.910194 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":187,"spinlock_wait_cycles":28544,"thread_start_us":411,"threads_started":1}
I20260929 01:05:52.911119 22381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135d668d938a47369d1260581337d550" candidate_uuid: "4e44c9db111b4433b5e2350a2553e17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee210061f714e0eb880707328129008" is_pre_election: true
I20260929 01:05:52.911621 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135d668d938a47369d1260581337d550" candidate_uuid: "4e44c9db111b4433b5e2350a2553e17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c9543fee6f41e2928f7cf1efd83211" is_pre_election: true
I20260929 01:05:52.911770 22381 raft_consensus.cc:2454] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e44c9db111b4433b5e2350a2553e17a in term 0.
I20260929 01:05:52.912127 22119 raft_consensus.cc:2454] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e44c9db111b4433b5e2350a2553e17a in term 0.
I20260929 01:05:52.912741 22186 leader_election.cc:302] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e44c9db111b4433b5e2350a2553e17a, cee210061f714e0eb880707328129008; no voters:
I20260929 01:05:52.913280 22451 raft_consensus.cc:2781] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:52.913453 22451 raft_consensus.cc:482] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:52.913635 22451 raft_consensus.cc:3037] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:52.916244 22451 raft_consensus.cc:504] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.916930 22451 leader_election.cc:288] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [CANDIDATE]: Term 1 election: Requested vote from peers 77c9543fee6f41e2928f7cf1efd83211 (127.3.21.193:45899), cee210061f714e0eb880707328129008 (127.3.21.195:46233)
I20260929 01:05:52.917620 22381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135d668d938a47369d1260581337d550" candidate_uuid: "4e44c9db111b4433b5e2350a2553e17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee210061f714e0eb880707328129008"
I20260929 01:05:52.917647 22119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135d668d938a47369d1260581337d550" candidate_uuid: "4e44c9db111b4433b5e2350a2553e17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77c9543fee6f41e2928f7cf1efd83211"
I20260929 01:05:52.917878 22119 raft_consensus.cc:3037] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:52.917879 22381 raft_consensus.cc:3037] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:52.921486 22381 raft_consensus.cc:2454] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e44c9db111b4433b5e2350a2553e17a in term 1.
I20260929 01:05:52.921486 22119 raft_consensus.cc:2454] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e44c9db111b4433b5e2350a2553e17a in term 1.
I20260929 01:05:52.922113 22186 leader_election.cc:302] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e44c9db111b4433b5e2350a2553e17a, 77c9543fee6f41e2928f7cf1efd83211, cee210061f714e0eb880707328129008; no voters:
I20260929 01:05:52.922616 22451 raft_consensus.cc:2781] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:52.923213 22451 raft_consensus.cc:686] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 1 LEADER]: Becoming Leader. State: Replica: 4e44c9db111b4433b5e2350a2553e17a, State: Running, Role: LEADER
I20260929 01:05:52.923787 22451 consensus_queue.cc:237] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } }
I20260929 01:05:52.929684 21977 catalog_manager.cc:5495] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e44c9db111b4433b5e2350a2553e17a (127.3.21.194). New cstate: current_term: 1 leader_uuid: "4e44c9db111b4433b5e2350a2553e17a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e44c9db111b4433b5e2350a2553e17a" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 38327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 } health_report { overall_health: UNKNOWN } } }
W20260929 01:05:52.951778 22165 tablet.cc:2369] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:52.970276 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.963551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.3.21.194:38327)
0929 01:05:52.963794 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:52.963800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:52.963821 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:05:52.963912 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:52.963916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:52.964019 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:52.964162 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:52.964537 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.965306 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.967181 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:52.967748 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:52.967879 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:52.968680 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:52.969337 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:52.969493 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:52.969777 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 01:05:52.969836 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 01:05:52.979959 22119 raft_consensus.cc:1261] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 1 FOLLOWER]: Refusing update from remote peer 4e44c9db111b4433b5e2350a2553e17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:52.980099 22381 raft_consensus.cc:1261] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 1 FOLLOWER]: Refusing update from remote peer 4e44c9db111b4433b5e2350a2553e17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 01:05:52.981027 22296 tablet.cc:2369] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:52.981243 22455 consensus_queue.cc:1035] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77c9543fee6f41e2928f7cf1efd83211" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 45899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:52.981688 22451 consensus_queue.cc:1035] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee210061f714e0eb880707328129008" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:05:52.986845 22427 tablet.cc:2369] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:52.996412 22458 mvcc.cc:205] Tried to move back new op lower bound from 7334477631393976320 to 7334477631180648448. Current Snapshot: MvccSnapshot[applied={T|T < 7334477631393976320}]
I20260929 01:05:52.997360 22459 mvcc.cc:205] Tried to move back new op lower bound from 7334477631393976320 to 7334477631180648448. Current Snapshot: MvccSnapshot[applied={T|T < 7334477631393976320}]
I20260929 01:05:53.015789 21978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:53.016124 21978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:53.020222 21978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22411af8676e44279d8a8fa3982e7d53: Sending DeleteTablet for 3 replicas of tablet 135d668d938a47369d1260581337d550
I20260929 01:05:53.021525 22099 tablet_service.cc:1514] Processing DeleteTablet for tablet 135d668d938a47369d1260581337d550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:53 UTC) from {username='slave'} at 127.0.0.1:35270
I20260929 01:05:53.021636 22361 tablet_service.cc:1514] Processing DeleteTablet for tablet 135d668d938a47369d1260581337d550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:53 UTC) from {username='slave'} at 127.0.0.1:34946
I20260929 01:05:53.021801 22230 tablet_service.cc:1514] Processing DeleteTablet for tablet 135d668d938a47369d1260581337d550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:53 UTC) from {username='slave'} at 127.0.0.1:38712
I20260929 01:05:53.024274 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22037
I20260929 01:05:53.034353 22469 tablet_replica.cc:330] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: stopping tablet replica
I20260929 01:05:53.035101 22469 raft_consensus.cc:2229] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:53.035728 22469 raft_consensus.cc:2258] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:53.038595 22469 ts_tablet_manager.cc:1905] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:53.047418 22468 tablet_replica.cc:330] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: stopping tablet replica
I20260929 01:05:53.047981 22468 raft_consensus.cc:2229] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:53.048698 22468 raft_consensus.cc:2258] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:53.049716 22469 ts_tablet_manager.cc:1918] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:53.049916 22469 log.cc:1196] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/135d668d938a47369d1260581337d550
I20260929 01:05:53.050712 22469 ts_tablet_manager.cc:1939] T 135d668d938a47369d1260581337d550 P cee210061f714e0eb880707328129008: Deleting consensus metadata
I20260929 01:05:53.052910 21965 catalog_manager.cc:4841] TS cee210061f714e0eb880707328129008 (127.3.21.195:46233): tablet 135d668d938a47369d1260581337d550 (table test-table [id=540c832df0db4190bba26e4934262ef9]) successfully deleted
I20260929 01:05:53.053045 22468 ts_tablet_manager.cc:1905] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:53.056751 22470 tablet_replica.cc:330] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: stopping tablet replica
I20260929 01:05:53.057296 22470 raft_consensus.cc:2229] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:53.058063 22470 raft_consensus.cc:2258] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:53.060446 22470 ts_tablet_manager.cc:1905] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:53.061969 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.026075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.3.21.193:45899)
0929 01:05:53.026292 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:53.026300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:53.046743 (+ 20443us) server_negotiation.cc:407] Connection header received
0929 01:05:53.047389 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:53.047404 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:53.047559 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:53.047646 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:53.048805 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:53.050875 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:53.057096 (+ 6221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:53.057832 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:53.057974 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:53.059860 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:53.059878 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:53.059897 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:53.060785 (+ 888us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:53.060858 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:53.060891 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:53.061003 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:53.061380 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 01:05:53.061529 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:05:53.068300 22470 ts_tablet_manager.cc:1918] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:53.068512 22470 log.cc:1196] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/135d668d938a47369d1260581337d550
I20260929 01:05:53.069134 22470 ts_tablet_manager.cc:1939] T 135d668d938a47369d1260581337d550 P 4e44c9db111b4433b5e2350a2553e17a: Deleting consensus metadata
I20260929 01:05:53.069660 22159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:53.069796 22468 ts_tablet_manager.cc:1918] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:53.070081 22468 log.cc:1196] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/135d668d938a47369d1260581337d550
I20260929 01:05:53.071058 21965 catalog_manager.cc:4841] TS 4e44c9db111b4433b5e2350a2553e17a (127.3.21.194:38327): tablet 135d668d938a47369d1260581337d550 (table test-table [id=540c832df0db4190bba26e4934262ef9]) successfully deleted
I20260929 01:05:53.195777 22468 ts_tablet_manager.cc:1939] T 135d668d938a47369d1260581337d550 P 77c9543fee6f41e2928f7cf1efd83211: Deleting consensus metadata
I20260929 01:05:53.195891 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22037
I20260929 01:05:53.214284 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22168
W20260929 01:05:53.214329 21963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.21.193:45899 (error 108)
I20260929 01:05:53.218057 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.217481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35294)
0929 01:05:53.217712 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.217852 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 01:05:53.218434 21963 catalog_manager.cc:4584] TS 77c9543fee6f41e2928f7cf1efd83211 (127.3.21.193:45899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 135d668d938a47369d1260581337d550: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
I20260929 01:05:53.222852 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.215607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.3.21.194:38327)
0929 01:05:53.215814 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:53.215823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:53.215851 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:05:53.215969 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:53.215976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:53.216109 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:53.216234 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:53.217033 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:53.218011 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:53.219558 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:53.219997 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:53.220150 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:53.221281 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:53.221290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:53.221306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:53.221940 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:53.221989 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:53.222009 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:53.222188 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:53.222336 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 01:05:53.222443 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:05:53.223610 22290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:53.272523 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.271904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35304)
0929 01:05:53.272146 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.272252 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 01:05:53.273751 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.273381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35306)
0929 01:05:53.273544 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.273620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:05:53.334880 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.334213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35312)
0929 01:05:53.334578 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.334702 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 01:05:53.336203 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.335699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35324)
0929 01:05:53.335872 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.335967 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 01:05:53.348575 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22168
I20260929 01:05:53.371300 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22299
I20260929 01:05:53.386441 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.372351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.3.21.195:46233)
0929 01:05:53.372553 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:53.372558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:53.372576 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:05:53.372776 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:53.372780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:53.372876 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:53.372986 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:53.373477 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:53.374323 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:53.375936 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:53.376599 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:53.376743 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:53.384677 (+ 7934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:53.384689 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:53.384706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:53.385365 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:53.385432 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:53.385462 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:53.385576 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:53.385834 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 01:05:53.385963 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 01:05:53.387454 22421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:53.417434 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.416865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35336)
0929 01:05:53.417122 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.417241 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 01:05:53.418682 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.418188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:45899 (local address 127.0.0.1:35346)
0929 01:05:53.418409 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.418562 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 01:05:53.480827 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22299
I20260929 01:05:53.499008 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 21947
I20260929 01:05:53.500123 22008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:53.615320 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 21947
2026-09-29T01:05:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3112 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:05:53 dist-test-slave-k0b7 krb5kdc[22479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:05:53 dist-test-slave-k0b7 krb5kdc[22479](info): set up 2 sockets
Sep 29 01:05:53 dist-test-slave-k0b7 krb5kdc[22479](info): commencing operation
krb5kdc: starting...
W20260929 01:05:54.811075 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:05:54 dist-test-slave-k0b7 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:05:55.108058 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:40199
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:55.486970 22492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:55.487380 22492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:55.487545 22492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:55.495729 22492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:55.495860 22492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:55.495968 22492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:55.496093 22492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:55.496181 22492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:55.506747 22492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:55.508227 22492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:55.509650 22492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:55.523682 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:55.523788 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:55.523748 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:55.524423 22492 server_base.cc:1034] running on GCE node
I20260929 01:05:55.525496 22492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:55.527227 22492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:55.528484 22492 hybrid_clock.cc:648] HybridClock initialized: now 1790643955528426 us; error 54 us; skew 500 ppm
I20260929 01:05:55.528949 22492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:55 dist-test-slave-k0b7 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643955, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:55.532651 22492 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:05:55.534441 22492 webserver.cc:460] Webserver started at http://127.3.21.254:42845/ using document root <none> and password file <none>
I20260929 01:05:55.535252 22492 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:55.535362 22492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:55.535646 22492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:55.538892 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "06e338e6df274eb79c84688c28c68e07"
format_stamp: "Formatted at 2026-09-29 01:05:55 on dist-test-slave-k0b7"
server_key: "551dce396a2d5e110ca4d06556683d94"
server_key_iv: "18a86eb0963181dbbc48430def43fe84"
server_key_version: "encryptionkey@0"
I20260929 01:05:55.539685 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "06e338e6df274eb79c84688c28c68e07"
format_stamp: "Formatted at 2026-09-29 01:05:55 on dist-test-slave-k0b7"
server_key: "551dce396a2d5e110ca4d06556683d94"
server_key_iv: "18a86eb0963181dbbc48430def43fe84"
server_key_version: "encryptionkey@0"
I20260929 01:05:55.545619 22492 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:05:55.549603 22507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:55.551277 22492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:05:55.551605 22492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "06e338e6df274eb79c84688c28c68e07"
format_stamp: "Formatted at 2026-09-29 01:05:55 on dist-test-slave-k0b7"
server_key: "551dce396a2d5e110ca4d06556683d94"
server_key_iv: "18a86eb0963181dbbc48430def43fe84"
server_key_version: "encryptionkey@0"
I20260929 01:05:55.551865 22492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:55.570209 22492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:55.575717 22492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:55.576062 22492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:55.599658 22492 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:40199
I20260929 01:05:55.599690 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:40199 every 8 connection(s)
I20260929 01:05:55.601173 22492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:55.605338 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:55.607748 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22492
I20260929 01:05:55.607935 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:05:55.608520 3159 external_mini_cluster.cc:1428] Setting key 7f37e4134007743b268efa4f7c4217be
I20260929 01:05:55.619277 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07: Bootstrap starting.
I20260929 01:05:55.623739 22559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:55.625156 22559 log.cc:824] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:05:55 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:55.628484 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07: No bootstrap required, opened a new log
I20260929 01:05:55.636615 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:55.610735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.3.21.254:40199)
0929 01:05:55.611749 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:55.611764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:55.611809 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:05:55.613633 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:55.613658 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:55.614111 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:55.614453 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:55.616547 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:55.617938 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:55.620077 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:55.620951 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:55.621159 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:55.627068 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:55.627077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:55.627094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:55.627136 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:55.630776 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:55.631650 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:55.631659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:55.631666 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:55.631862 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:55.632340 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:55.632351 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:55.632353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:55.633017 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:55.633412 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:55.633905 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 01:05:55.634413 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":536,"threads_started":1}
I20260929 01:05:55.638747 22559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e338e6df274eb79c84688c28c68e07" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 40199 } }
I20260929 01:05:55.639201 22559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:55.639325 22559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e338e6df274eb79c84688c28c68e07, State: Initialized, Role: FOLLOWER
I20260929 01:05:55.639914 22559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e338e6df274eb79c84688c28c68e07" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 40199 } }
I20260929 01:05:55.640193 22559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:55.640353 22559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:55.640561 22559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:55.643536 22559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e338e6df274eb79c84688c28c68e07" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 40199 } }
I20260929 01:05:55.644068 22559 leader_election.cc:302] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e338e6df274eb79c84688c28c68e07; no voters:
I20260929 01:05:55.644918 22559 leader_election.cc:288] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:55.645183 22564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:55.646020 22564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [term 1 LEADER]: Becoming Leader. State: Replica: 06e338e6df274eb79c84688c28c68e07, State: Running, Role: LEADER
I20260929 01:05:55.646755 22564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e338e6df274eb79c84688c28c68e07" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 40199 } }
I20260929 01:05:55.647804 22559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:55.650477 22566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e338e6df274eb79c84688c28c68e07. Latest consensus state: current_term: 1 leader_uuid: "06e338e6df274eb79c84688c28c68e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e338e6df274eb79c84688c28c68e07" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 40199 } } }
I20260929 01:05:55.650764 22565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e338e6df274eb79c84688c28c68e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e338e6df274eb79c84688c28c68e07" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 40199 } } }
I20260929 01:05:55.651042 22566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:55.651124 22565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:55.657838 22573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:55.664304 22573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:55.675074 22573 catalog_manager.cc:1347] Generated new cluster ID: fcf4d4082500411dabe5fb70649dab7e
I20260929 01:05:55.675225 22573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:55.684836 22573 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:55.687026 22573 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:55.696825 22573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07: Generated new TSK 0
I20260929 01:05:55.698065 22573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.21.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
Entry for principal kudu/127.3.21.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab.
I20260929 01:05:55.777603 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:40199
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:56.172616 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:56.172930 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:56.173094 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:56.180938 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:56.181083 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:56.181275 22585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:56.192253 22585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.193.keytab
--principal=kudu/127.3.21.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:40199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:56.193670 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:56.195127 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:56.209695 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:56.209618 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:56.209667 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:56.210668 22585 server_base.cc:1034] running on GCE node
I20260929 01:05:56.211577 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:56.212773 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:56.214149 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790643956213970 us; error 82 us; skew 500 ppm
I20260929 01:05:56.214735 22585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:56 dist-test-slave-k0b7 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:56.218827 22585 init.cc:377] Logged in from keytab as kudu/127.3.21.193@KRBTEST.COM (short username kudu)
I20260929 01:05:56.220741 22585 webserver.cc:460] Webserver started at http://127.3.21.193:46815/ using document root <none> and password file <none>
I20260929 01:05:56.221572 22585 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:56.221704 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:56.222016 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:56.225662 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "11726aa186404cdcbe799f55877455ee"
format_stamp: "Formatted at 2026-09-29 01:05:56 on dist-test-slave-k0b7"
server_key: "4054f1d4e932a8c7fee5b2e4b5fce9aa"
server_key_iv: "9ce3b75ca69857246c7ed6ab91748fd9"
server_key_version: "encryptionkey@0"
I20260929 01:05:56.226780 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "11726aa186404cdcbe799f55877455ee"
format_stamp: "Formatted at 2026-09-29 01:05:56 on dist-test-slave-k0b7"
server_key: "4054f1d4e932a8c7fee5b2e4b5fce9aa"
server_key_iv: "9ce3b75ca69857246c7ed6ab91748fd9"
server_key_version: "encryptionkey@0"
I20260929 01:05:56.233705 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260929 01:05:56.238260 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:56.240378 22585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:05:56.240620 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "11726aa186404cdcbe799f55877455ee"
format_stamp: "Formatted at 2026-09-29 01:05:56 on dist-test-slave-k0b7"
server_key: "4054f1d4e932a8c7fee5b2e4b5fce9aa"
server_key_iv: "9ce3b75ca69857246c7ed6ab91748fd9"
server_key_version: "encryptionkey@0"
I20260929 01:05:56.240876 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:56.260598 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:56.266647 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:56.267048 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:56.268468 22585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:56.270182 22585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:56.270282 22585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:56.270548 22585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:56.270648 22585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 01:05:56.316037 22585 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:36923
I20260929 01:05:56.316068 22712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:36923 every 8 connection(s)
I20260929 01:05:56.317467 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:56.319074 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22585
I20260929 01:05:56.319365 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:56.320099 3159 external_mini_cluster.cc:1428] Setting key 6a7edbfec31882edd4cf98ce9fd6c380
Sep 29 01:05:56 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.193@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:56.346096 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.321410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:32801 (local address 127.3.21.254:40199)
0929 01:05:56.321919 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:56.321926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:56.323487 (+ 1561us) server_negotiation.cc:407] Connection header received
0929 01:05:56.326634 (+ 3147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:56.326640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:56.326727 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:56.326830 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:56.329214 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.329982 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.332865 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.333510 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.333734 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:56.339375 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:56.339381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:56.339384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:56.339424 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:56.342599 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:56.343527 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:56.343538 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:56.343540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:56.343675 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:56.344192 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:56.344196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:56.344199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:56.344485 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:56.344651 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:56.345445 (+ 794us) server_negotiation.cc:299] Negotiation successful
0929 01:05:56.345708 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260929 01:05:56.347283 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.322128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:40199 (local address 127.3.21.193:32801)
0929 01:05:56.323285 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:56.323343 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:56.326217 (+ 2874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:56.327155 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:56.327179 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:56.328032 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:56.328952 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:56.328986 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.330195 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.330203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:56.332592 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:56.332618 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.333863 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.333873 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:56.334324 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:56.335489 (+ 1165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:56.335535 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:56.339042 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:56.342809 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:56.342818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:56.342835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:56.343369 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:56.343798 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:56.343803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:56.343817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:56.344047 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:56.344622 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:56.344632 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:56.345036 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 01:05:56.345325 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 01:05:56.345930 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":405,"threads_started":1}
I20260929 01:05:56.349377 22713 heartbeater.cc:347] Connected to a master server at 127.3.21.254:40199
I20260929 01:05:56.349720 22713 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:56.350585 22713 heartbeater.cc:511] Master 127.3.21.254:40199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.21.194@KRBTEST.COM; defaulting to no policy
I20260929 01:05:56.353502 22524 ts_manager.cc:194] Registered new tserver with Master: 11726aa186404cdcbe799f55877455ee (127.3.21.193:36923)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.194@KRBTEST.COM" created.
I20260929 01:05:56.355465 22524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.193@KRBTEST.COM'} at 127.3.21.193:32801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
Entry for principal kudu/127.3.21.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab.
I20260929 01:05:56.391531 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:40199
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:56.751763 22719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:56.752072 22719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:56.752213 22719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:56.760196 22719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:56.760326 22719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:56.760501 22719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:05:56.774019 22719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.194.keytab
--principal=kudu/127.3.21.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:40199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:56.775689 22719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:56.777468 22719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:56.794621 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:56.794674 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:56.794600 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:56.795864 22719 server_base.cc:1034] running on GCE node
I20260929 01:05:56.796649 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:56.797879 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:56.799373 22719 hybrid_clock.cc:648] HybridClock initialized: now 1790643956799327 us; error 56 us; skew 500 ppm
I20260929 01:05:56.799798 22719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:56 dist-test-slave-k0b7 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:56.804018 22719 init.cc:377] Logged in from keytab as kudu/127.3.21.194@KRBTEST.COM (short username kudu)
I20260929 01:05:56.806432 22719 webserver.cc:460] Webserver started at http://127.3.21.194:38453/ using document root <none> and password file <none>
I20260929 01:05:56.807403 22719 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:56.807605 22719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:56.808033 22719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:56.812146 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "6f621e25827b4f0f890c0085ec5d6085"
format_stamp: "Formatted at 2026-09-29 01:05:56 on dist-test-slave-k0b7"
server_key: "0c33ba46a298176980c97e00ad69937a"
server_key_iv: "a3f58389740699a25b4874d3864fac9b"
server_key_version: "encryptionkey@0"
I20260929 01:05:56.813215 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "6f621e25827b4f0f890c0085ec5d6085"
format_stamp: "Formatted at 2026-09-29 01:05:56 on dist-test-slave-k0b7"
server_key: "0c33ba46a298176980c97e00ad69937a"
server_key_iv: "a3f58389740699a25b4874d3864fac9b"
server_key_version: "encryptionkey@0"
I20260929 01:05:56.820003 22719 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:05:56.824654 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:56.826375 22719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:05:56.826781 22719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "6f621e25827b4f0f890c0085ec5d6085"
format_stamp: "Formatted at 2026-09-29 01:05:56 on dist-test-slave-k0b7"
server_key: "0c33ba46a298176980c97e00ad69937a"
server_key_iv: "a3f58389740699a25b4874d3864fac9b"
server_key_version: "encryptionkey@0"
I20260929 01:05:56.827029 22719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:56.853587 22719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:56.858790 22719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:56.859171 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:56.860324 22719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:56.861934 22719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:56.862017 22719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:56.862175 22719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:56.862231 22719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:56.906247 22719 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:34265
I20260929 01:05:56.906318 22846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:34265 every 8 connection(s)
I20260929 01:05:56.907879 22719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:56.911952 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22719
I20260929 01:05:56.912278 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:05:56.912798 3159 external_mini_cluster.cc:1428] Setting key 2619906c88b23d43aae3542a8743b950
Sep 29 01:05:56 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643956, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.194@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:56.939819 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.911224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:46199 (local address 127.3.21.254:40199)
0929 01:05:56.911818 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:56.911826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:56.912833 (+ 1007us) server_negotiation.cc:407] Connection header received
0929 01:05:56.915390 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:56.915399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:56.915505 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:56.915660 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:56.918983 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.920104 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.923659 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.924223 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.924369 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:56.930382 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:56.930394 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:56.930402 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:56.930464 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:56.935151 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:56.936340 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:56.936350 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:56.936357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:56.936585 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:56.937172 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:56.937181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:56.937187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:56.937686 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:56.937896 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:56.938925 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0929 01:05:56.939251 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":360,"threads_started":1}
I20260929 01:05:56.940598 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.911614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:40199 (local address 127.3.21.194:46199)
0929 01:05:56.912582 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:56.912654 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:56.914818 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:56.916344 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:56.916361 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:56.917426 (+ 1065us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:56.918672 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:56.918713 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.920356 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.920366 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:56.923285 (+ 2919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:56.923319 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:56.924590 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:56.924604 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:56.925100 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:56.926281 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:56.926332 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:56.929977 (+ 3645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:56.935402 (+ 5425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:56.935412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:56.935429 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:56.936142 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:56.936782 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:56.936787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:56.936789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:56.937009 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:56.937916 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:56.937923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:56.938367 (+ 444us) client_negotiation.cc:769] Sending connection context
0929 01:05:56.938755 (+ 388us) client_negotiation.cc:240] Negotiation successful
0929 01:05:56.939261 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":407,"threads_started":1}
I20260929 01:05:56.942889 22847 heartbeater.cc:347] Connected to a master server at 127.3.21.254:40199
I20260929 01:05:56.943320 22847 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:56.944458 22847 heartbeater.cc:511] Master 127.3.21.254:40199 requested a full tablet report, sending...
I20260929 01:05:56.946751 22524 ts_manager.cc:194] Registered new tserver with Master: 6f621e25827b4f0f890c0085ec5d6085 (127.3.21.194:34265)
WARNING: no policy specified for kudu/127.3.21.195@KRBTEST.COM; defaulting to no policy
I20260929 01:05:56.947839 22524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.194@KRBTEST.COM'} at 127.3.21.194:46199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
Entry for principal kudu/127.3.21.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab.
I20260929 01:05:56.986950 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:40199
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:57.351248 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:57.351625 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:57.351768 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:05:57.359789 22713 heartbeater.cc:503] Master 127.3.21.254:40199 was elected leader, sending a full tablet report...
W20260929 01:05:57.360464 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:57.360622 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:57.360813 22853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:05:57.371534 22853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.195.keytab
--principal=kudu/127.3.21.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:40199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:57.372972 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:57.374429 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:57.389461 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:57.389497 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:57.389462 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:57.390996 22853 server_base.cc:1034] running on GCE node
I20260929 01:05:57.391680 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:57.392921 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:57.394210 22853 hybrid_clock.cc:648] HybridClock initialized: now 1790643957394153 us; error 60 us; skew 500 ppm
I20260929 01:05:57.394678 22853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:05:57 dist-test-slave-k0b7 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:57.398779 22853 init.cc:377] Logged in from keytab as kudu/127.3.21.195@KRBTEST.COM (short username kudu)
I20260929 01:05:57.400548 22853 webserver.cc:460] Webserver started at http://127.3.21.195:42591/ using document root <none> and password file <none>
I20260929 01:05:57.401394 22853 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:57.401553 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:57.401934 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:57.405162 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "8327034a93284c568d7684ae3c5a50f7"
format_stamp: "Formatted at 2026-09-29 01:05:57 on dist-test-slave-k0b7"
server_key: "cb63c044eeaa13f7bc64ed3fc8922247"
server_key_iv: "bd59b154430f5633efecb627f5b8b53e"
server_key_version: "encryptionkey@0"
I20260929 01:05:57.406041 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "8327034a93284c568d7684ae3c5a50f7"
format_stamp: "Formatted at 2026-09-29 01:05:57 on dist-test-slave-k0b7"
server_key: "cb63c044eeaa13f7bc64ed3fc8922247"
server_key_iv: "bd59b154430f5633efecb627f5b8b53e"
server_key_version: "encryptionkey@0"
I20260929 01:05:57.412266 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260929 01:05:57.416687 22868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:57.419112 22853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:05:57.419358 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "8327034a93284c568d7684ae3c5a50f7"
format_stamp: "Formatted at 2026-09-29 01:05:57 on dist-test-slave-k0b7"
server_key: "cb63c044eeaa13f7bc64ed3fc8922247"
server_key_iv: "bd59b154430f5633efecb627f5b8b53e"
server_key_version: "encryptionkey@0"
I20260929 01:05:57.419623 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:57.440781 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:57.446213 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:57.446627 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:57.447958 22853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:57.449807 22853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:57.449921 22853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:57.450227 22853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:57.450364 22853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:57.490691 22853 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:34303
I20260929 01:05:57.490705 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:34303 every 8 connection(s)
I20260929 01:05:57.492230 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:57.497881 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 22853
I20260929 01:05:57.498049 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:05:57.498687 3159 external_mini_cluster.cc:1428] Setting key e149ea6ec48039dd964ec715e2b8086d
Sep 29 01:05:57 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643957, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.195@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:05:57.518117 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.495441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:41325 (local address 127.3.21.254:40199)
0929 01:05:57.495949 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.495954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.497383 (+ 1429us) server_negotiation.cc:407] Connection header received
0929 01:05:57.500698 (+ 3315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.500703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.500782 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.500919 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:57.503217 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.504116 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.505989 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.506467 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.506688 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.510987 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:57.510993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:57.510997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:57.511046 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:57.514361 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:57.515347 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:57.515354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:57.515358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:57.515506 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:57.516038 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:57.516046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:57.516055 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:57.516356 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:57.516599 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.517405 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.517677 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":282,"threads_started":1}
I20260929 01:05:57.519299 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.496061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:40199 (local address 127.3.21.195:41325)
0929 01:05:57.497176 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.497237 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.500340 (+ 3103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.501284 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.501300 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.502217 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:57.503018 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.503040 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.504299 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.504306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.505768 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.505784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.506829 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.506838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.507119 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.507936 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:57.507969 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:57.510690 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:57.514605 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:57.514624 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:57.514643 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:57.515189 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:57.515651 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:57.515660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:57.515665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:57.515895 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:57.516512 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:57.516523 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:57.516973 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.517291 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.517941 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":516,"threads_started":1}
I20260929 01:05:57.520973 22981 heartbeater.cc:347] Connected to a master server at 127.3.21.254:40199
I20260929 01:05:57.521406 22981 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:57.522214 22981 heartbeater.cc:511] Master 127.3.21.254:40199 requested a full tablet report, sending...
I20260929 01:05:57.524155 22524 ts_manager.cc:194] Registered new tserver with Master: 8327034a93284c568d7684ae3c5a50f7 (127.3.21.195:34303)
I20260929 01:05:57.525125 22524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.21.195@KRBTEST.COM'} at 127.3.21.195:41325
I20260929 01:05:57.528663 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:57.549153 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.533852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.3.21.254:40199)
0929 01:05:57.534081 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.534089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.534456 (+ 367us) server_negotiation.cc:407] Connection header received
0929 01:05:57.534726 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.534746 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.534822 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.534941 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:57.536555 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.537387 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.538985 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.539461 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.539628 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.542789 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:57.542795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:57.542799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:57.542847 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:57.545961 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:57.546828 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:57.546853 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:57.546859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:57.547018 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:57.547428 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:57.547433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:57.547437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:57.547879 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:57.548160 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.548530 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.548796 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:05:57.553684 22524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:57.561342 22524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:57.600777 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.575318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.3.21.194:34265)
0929 01:05:57.587569 (+ 12251us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.587582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.589692 (+ 2110us) server_negotiation.cc:407] Connection header received
0929 01:05:57.589836 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.589849 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.590218 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.590450 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.591969 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.594309 (+ 2340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.597951 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.599419 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.599586 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.599830 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.600248 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.600435 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11986,"thread_start_us":12928,"threads_started":1}
I20260929 01:05:57.600903 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.575102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:34265 (local address 127.0.0.1:49876)
0929 01:05:57.589119 (+ 14017us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.589181 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.589418 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.591057 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.591070 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.591119 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.591652 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.591670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.594521 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.594528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.597733 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.597748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.599671 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.599677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.599993 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.600032 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.600176 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.600391 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13719,"mutex_wait_us":1,"thread_start_us":21064,"threads_started":1}
I20260929 01:05:57.605716 22782 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef332454558401aa69d64c8c710e0f6 (DEFAULT_TABLE table=test-table [id=6b662b0d83d247329fdc753ea95dc349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:57.607962 22782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef332454558401aa69d64c8c710e0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:57.608018 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.575341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52596)
0929 01:05:57.593878 (+ 18537us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.593901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.594037 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.595465 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.595477 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.595522 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.596493 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.596506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.598579 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.598584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.602857 (+ 4273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.602873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.605254 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.605261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.605464 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.605503 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.605598 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.607527 (+ 1929us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18384,"mutex_wait_us":1516,"thread_start_us":13330,"threads_started":1}
I20260929 01:05:57.608557 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.576126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.3.21.193:36923)
0929 01:05:57.594711 (+ 18585us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.594724 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.594773 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:05:57.594908 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.594916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.595213 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.595446 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.596685 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.598383 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.603057 (+ 4674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.604585 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.604809 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.605072 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.607165 (+ 2093us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.608248 (+ 1083us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18339,"thread_start_us":19520,"threads_started":1}
I20260929 01:05:57.608757 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.576626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.3.21.195:34303)
0929 01:05:57.588207 (+ 11581us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.588219 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.596762 (+ 8543us) server_negotiation.cc:407] Connection header received
0929 01:05:57.596940 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.596949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.597227 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.597432 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.598561 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.601193 (+ 2632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.605586 (+ 4393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.607211 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.607321 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.607558 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.608194 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.608380 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11331,"thread_start_us":11161,"threads_started":1}
I20260929 01:05:57.609464 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.575840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:34303 (local address 127.0.0.1:60074)
0929 01:05:57.596576 (+ 20736us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.596596 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.596787 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.597710 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.597715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.597743 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.598268 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.598283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.603035 (+ 4752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.603040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.605264 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.605278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.607773 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.607780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.607983 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.608021 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.608977 (+ 956us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.609142 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20588,"thread_start_us":17704,"threads_started":1}
I20260929 01:05:57.614001 22648 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef332454558401aa69d64c8c710e0f6 (DEFAULT_TABLE table=test-table [id=6b662b0d83d247329fdc753ea95dc349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:57.613786 22916 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef332454558401aa69d64c8c710e0f6 (DEFAULT_TABLE table=test-table [id=6b662b0d83d247329fdc753ea95dc349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:57.615526 22916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef332454558401aa69d64c8c710e0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:57.615526 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef332454558401aa69d64c8c710e0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:57.630991 23000 tablet_bootstrap.cc:492] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Bootstrap starting.
I20260929 01:05:57.632103 23001 tablet_bootstrap.cc:492] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Bootstrap starting.
I20260929 01:05:57.633111 23002 tablet_bootstrap.cc:492] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Bootstrap starting.
I20260929 01:05:57.635344 23000 tablet_bootstrap.cc:654] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:57.636755 23000 log.cc:824] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:57.636920 23002 tablet_bootstrap.cc:654] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:57.637639 23001 tablet_bootstrap.cc:654] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:57.638268 23002 log.cc:824] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:57.638921 23001 log.cc:824] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:57.640588 23000 tablet_bootstrap.cc:492] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: No bootstrap required, opened a new log
I20260929 01:05:57.640915 23000 ts_tablet_manager.cc:1397] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260929 01:05:57.641521 23002 tablet_bootstrap.cc:492] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: No bootstrap required, opened a new log
I20260929 01:05:57.641798 23002 ts_tablet_manager.cc:1397] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260929 01:05:57.642855 23001 tablet_bootstrap.cc:492] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: No bootstrap required, opened a new log
I20260929 01:05:57.643115 23001 ts_tablet_manager.cc:1397] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 01:05:57.651798 23001 raft_consensus.cc:348] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.652307 23001 raft_consensus.cc:374] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:57.652405 23001 raft_consensus.cc:729] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11726aa186404cdcbe799f55877455ee, State: Initialized, Role: FOLLOWER
I20260929 01:05:57.652374 23000 raft_consensus.cc:348] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.652827 23000 raft_consensus.cc:374] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:57.652931 23000 raft_consensus.cc:729] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f621e25827b4f0f890c0085ec5d6085, State: Initialized, Role: FOLLOWER
I20260929 01:05:57.653146 23001 consensus_queue.cc:260] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.653012 23002 raft_consensus.cc:348] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.653455 23002 raft_consensus.cc:374] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:57.653565 23002 raft_consensus.cc:729] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8327034a93284c568d7684ae3c5a50f7, State: Initialized, Role: FOLLOWER
I20260929 01:05:57.653690 23000 consensus_queue.cc:260] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.654142 23002 consensus_queue.cc:260] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.655109 22847 heartbeater.cc:503] Master 127.3.21.254:40199 was elected leader, sending a full tablet report...
I20260929 01:05:57.655766 23000 ts_tablet_manager.cc:1428] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260929 01:05:57.657498 23001 ts_tablet_manager.cc:1428] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260929 01:05:57.659045 22981 heartbeater.cc:503] Master 127.3.21.254:40199 was elected leader, sending a full tablet report...
W20260929 01:05:57.660396 22848 tablet.cc:2369] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:57.663214 23002 ts_tablet_manager.cc:1428] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Time spent starting tablet: real 0.021s user 0.017s sys 0.000s
W20260929 01:05:57.744385 22982 tablet.cc:2369] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:57.768797 23006 raft_consensus.cc:482] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:57.769151 23006 raft_consensus.cc:504] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.771169 23006 leader_election.cc:288] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f621e25827b4f0f890c0085ec5d6085 (127.3.21.194:34265), 8327034a93284c568d7684ae3c5a50f7 (127.3.21.195:34303)
I20260929 01:05:57.772346 23007 raft_consensus.cc:482] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:57.772668 23007 raft_consensus.cc:504] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.780961 23007 leader_election.cc:288] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11726aa186404cdcbe799f55877455ee (127.3.21.193:36923), 8327034a93284c568d7684ae3c5a50f7 (127.3.21.195:34303)
I20260929 01:05:57.784538 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.771653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:59601 (local address 127.3.21.195:34303)
0929 01:05:57.771856 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.771864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.772882 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 01:05:57.773233 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.773240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.773377 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.773484 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.774403 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.776617 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.780719 (+ 4102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.782472 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.782735 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.782862 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.783828 (+ 966us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.784076 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 01:05:57.785943 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.771480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:34303 (local address 127.3.21.193:59601)
0929 01:05:57.772748 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.772769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.773025 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.773628 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.773634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.773680 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.774192 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.774210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.776918 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.776928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.780378 (+ 3450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.780414 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.783252 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.783259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.783511 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.783579 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.785232 (+ 1653us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.785428 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":1,"thread_start_us":482,"threads_started":1}
I20260929 01:05:57.787365 22936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef332454558401aa69d64c8c710e0f6" candidate_uuid: "11726aa186404cdcbe799f55877455ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8327034a93284c568d7684ae3c5a50f7" is_pre_election: true
I20260929 01:05:57.788043 22936 raft_consensus.cc:2454] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11726aa186404cdcbe799f55877455ee in term 0.
I20260929 01:05:57.789101 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.771698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:48555 (local address 127.3.21.194:34265)
0929 01:05:57.772039 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.772045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.773422 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 01:05:57.773609 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.773614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.773752 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.773855 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.774590 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.776099 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.786013 (+ 9914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.787693 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.787841 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.787916 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.788612 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.788808 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260929 01:05:57.789131 22604 leader_election.cc:302] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11726aa186404cdcbe799f55877455ee, 8327034a93284c568d7684ae3c5a50f7; no voters:
I20260929 01:05:57.789744 23006 raft_consensus.cc:2781] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:57.789856 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.771785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:34265 (local address 127.3.21.193:48555)
0929 01:05:57.773276 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.773301 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.773466 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.773901 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.773906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.773928 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.774394 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.774406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.780804 (+ 6398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.780813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.785417 (+ 4604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.785435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.788164 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.788172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.788366 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.788415 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.789293 (+ 878us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.789457 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"thread_start_us":994,"threads_started":1}
I20260929 01:05:57.789929 23006 raft_consensus.cc:482] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:57.790042 23006 raft_consensus.cc:3037] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:57.792014 22802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef332454558401aa69d64c8c710e0f6" candidate_uuid: "11726aa186404cdcbe799f55877455ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f621e25827b4f0f890c0085ec5d6085" is_pre_election: true
I20260929 01:05:57.792558 22802 raft_consensus.cc:2454] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11726aa186404cdcbe799f55877455ee in term 0.
I20260929 01:05:57.794516 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.776563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:33697 (local address 127.3.21.193:36923)
0929 01:05:57.776822 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.776831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.785806 (+ 8975us) server_negotiation.cc:407] Connection header received
0929 01:05:57.786145 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.786153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.786234 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.786302 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.787341 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.788394 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.791868 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.793174 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.793332 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.793508 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.794051 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.794216 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:05:57.794603 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.775207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.3.21.194:33697)
0929 01:05:57.785667 (+ 10460us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.785696 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.785939 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.786519 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.786525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.786560 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.787173 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.787185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.788583 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.788588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.791695 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.791709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.793668 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.793674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.793844 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.793879 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.793965 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.794109 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10285,"mutex_wait_us":20,"thread_start_us":10478,"threads_started":1}
I20260929 01:05:57.794710 23006 raft_consensus.cc:504] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.795542 23006 leader_election.cc:288] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [CANDIDATE]: Term 1 election: Requested vote from peers 6f621e25827b4f0f890c0085ec5d6085 (127.3.21.194:34265), 8327034a93284c568d7684ae3c5a50f7 (127.3.21.195:34303)
I20260929 01:05:57.795543 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef332454558401aa69d64c8c710e0f6" candidate_uuid: "6f621e25827b4f0f890c0085ec5d6085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11726aa186404cdcbe799f55877455ee" is_pre_election: true
I20260929 01:05:57.795945 22802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef332454558401aa69d64c8c710e0f6" candidate_uuid: "11726aa186404cdcbe799f55877455ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f621e25827b4f0f890c0085ec5d6085"
I20260929 01:05:57.796061 22668 raft_consensus.cc:2379] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f621e25827b4f0f890c0085ec5d6085 in current term 1: Already voted for candidate 11726aa186404cdcbe799f55877455ee in this term.
I20260929 01:05:57.796223 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.775414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:46911 (local address 127.3.21.195:34303)
0929 01:05:57.785179 (+ 9765us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.785185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.786173 (+ 988us) server_negotiation.cc:407] Connection header received
0929 01:05:57.786412 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.786419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.786549 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.786618 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:57.787682 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.789203 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.792296 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.793874 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.794054 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.794155 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.795738 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.795894 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9626,"thread_start_us":9602,"threads_started":1}
I20260929 01:05:57.796165 22802 raft_consensus.cc:3037] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:57.796124 22936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef332454558401aa69d64c8c710e0f6" candidate_uuid: "11726aa186404cdcbe799f55877455ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8327034a93284c568d7684ae3c5a50f7"
I20260929 01:05:57.796392 22936 raft_consensus.cc:3037] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:57.798815 22802 raft_consensus.cc:2454] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11726aa186404cdcbe799f55877455ee in term 1.
I20260929 01:05:57.799084 22936 raft_consensus.cc:2454] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11726aa186404cdcbe799f55877455ee in term 1.
I20260929 01:05:57.799296 22603 leader_election.cc:302] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11726aa186404cdcbe799f55877455ee, 6f621e25827b4f0f890c0085ec5d6085; no voters:
I20260929 01:05:57.799631 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.775211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:34303 (local address 127.3.21.194:46911)
0929 01:05:57.786080 (+ 10869us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.786098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:57.786260 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:57.786811 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:57.786817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:57.786852 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:57.787492 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.787504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.789393 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.789398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.792042 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:57.792058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.794370 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.794376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:57.795196 (+ 820us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.795224 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:05:57.795348 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 01:05:57.795497 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10718,"mutex_wait_us":153,"thread_start_us":10042,"threads_started":1}
I20260929 01:05:57.799678 23006 raft_consensus.cc:2781] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:57.800359 22936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef332454558401aa69d64c8c710e0f6" candidate_uuid: "6f621e25827b4f0f890c0085ec5d6085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8327034a93284c568d7684ae3c5a50f7" is_pre_election: true
I20260929 01:05:57.800735 22936 raft_consensus.cc:2379] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f621e25827b4f0f890c0085ec5d6085 in current term 1: Already voted for candidate 11726aa186404cdcbe799f55877455ee in this term.
I20260929 01:05:57.801296 23006 raft_consensus.cc:686] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 1 LEADER]: Becoming Leader. State: Replica: 11726aa186404cdcbe799f55877455ee, State: Running, Role: LEADER
I20260929 01:05:57.801601 22738 leader_election.cc:302] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f621e25827b4f0f890c0085ec5d6085; no voters: 11726aa186404cdcbe799f55877455ee, 8327034a93284c568d7684ae3c5a50f7
I20260929 01:05:57.801888 23006 consensus_queue.cc:237] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } }
I20260929 01:05:57.802175 23007 raft_consensus.cc:2726] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:05:57.807417 22523 catalog_manager.cc:5495] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 11726aa186404cdcbe799f55877455ee (127.3.21.193). New cstate: current_term: 1 leader_uuid: "11726aa186404cdcbe799f55877455ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11726aa186404cdcbe799f55877455ee" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 36923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 } health_report { overall_health: UNKNOWN } } }
W20260929 01:05:57.820464 22714 tablet.cc:2369] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:57.853318 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.846185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.3.21.193:36923)
0929 01:05:57.846428 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.846434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.846617 (+ 183us) server_negotiation.cc:407] Connection header received
0929 01:05:57.846861 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.846866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.846951 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.847112 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:57.847663 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.848754 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.850298 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.850863 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.850982 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.851757 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:57.852488 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:57.852649 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.852829 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.852958 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:05:57.862897 22936 raft_consensus.cc:1261] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 1 FOLLOWER]: Refusing update from remote peer 11726aa186404cdcbe799f55877455ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:57.862911 22802 raft_consensus.cc:1261] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 1 FOLLOWER]: Refusing update from remote peer 11726aa186404cdcbe799f55877455ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:57.863847 23006 consensus_queue.cc:1035] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f621e25827b4f0f890c0085ec5d6085" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 34265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:57.864161 23014 consensus_queue.cc:1035] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8327034a93284c568d7684ae3c5a50f7" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:57.874800 23017 mvcc.cc:205] Tried to move back new op lower bound from 7334477651394301952 to 7334477651161518080. Current Snapshot: MvccSnapshot[applied={T|T < 7334477651394301952}]
I20260929 01:05:57.877646 23018 mvcc.cc:205] Tried to move back new op lower bound from 7334477651394301952 to 7334477651161518080. Current Snapshot: MvccSnapshot[applied={T|T < 7334477651394301952}]
I20260929 01:05:57.896299 22522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:57.896677 22522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:57.901257 22522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06e338e6df274eb79c84688c28c68e07: Sending DeleteTablet for 3 replicas of tablet 9ef332454558401aa69d64c8c710e0f6
I20260929 01:05:57.902696 22648 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef332454558401aa69d64c8c710e0f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:57 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:52596
I20260929 01:05:57.902817 22782 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef332454558401aa69d64c8c710e0f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:57 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:49876
I20260929 01:05:57.903756 22916 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef332454558401aa69d64c8c710e0f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:57 UTC) from {username='kudu', principal='kudu/127.3.21.254@KRBTEST.COM'} at 127.0.0.1:60074
I20260929 01:05:57.904192 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22585
I20260929 01:05:57.904459 23028 tablet_replica.cc:330] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: stopping tablet replica
I20260929 01:05:57.905112 23028 raft_consensus.cc:2229] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:57.905997 23028 raft_consensus.cc:2258] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:57.909149 23028 ts_tablet_manager.cc:1905] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:57.918324 23029 tablet_replica.cc:330] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: stopping tablet replica
I20260929 01:05:57.918867 23029 raft_consensus.cc:2229] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:57.919423 23029 raft_consensus.cc:2258] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:57.923141 23028 ts_tablet_manager.cc:1918] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:57.923555 23028 log.cc:1196] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/9ef332454558401aa69d64c8c710e0f6
I20260929 01:05:57.924600 23028 ts_tablet_manager.cc:1939] T 9ef332454558401aa69d64c8c710e0f6 P 6f621e25827b4f0f890c0085ec5d6085: Deleting consensus metadata
I20260929 01:05:57.927426 23029 ts_tablet_manager.cc:1905] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:57.928686 22510 catalog_manager.cc:4841] TS 6f621e25827b4f0f890c0085ec5d6085 (127.3.21.194:34265): tablet 9ef332454558401aa69d64c8c710e0f6 (table test-table [id=6b662b0d83d247329fdc753ea95dc349]) successfully deleted
Sep 29 01:05:57 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.193@KRBTEST.COM
I20260929 01:05:57.938759 23027 tablet_replica.cc:330] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: stopping tablet replica
I20260929 01:05:57.938800 23029 ts_tablet_manager.cc:1918] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:57.939210 23029 log.cc:1196] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/9ef332454558401aa69d64c8c710e0f6
I20260929 01:05:57.939265 23027 raft_consensus.cc:2229] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:57.939915 23029 ts_tablet_manager.cc:1939] T 9ef332454558401aa69d64c8c710e0f6 P 8327034a93284c568d7684ae3c5a50f7: Deleting consensus metadata
I20260929 01:05:57.940068 23027 raft_consensus.cc:2258] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:57.943076 22511 catalog_manager.cc:4841] TS 8327034a93284c568d7684ae3c5a50f7 (127.3.21.195:34303): tablet 9ef332454558401aa69d64c8c710e0f6 (table test-table [id=6b662b0d83d247329fdc753ea95dc349]) successfully deleted
I20260929 01:05:57.945070 23027 ts_tablet_manager.cc:1905] T 9ef332454558401aa69d64c8c710e0f6 P 11726aa186404cdcbe799f55877455ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:57.949568 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.905291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.3.21.193:36923)
0929 01:05:57.905468 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:57.905475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:57.914058 (+ 8583us) server_negotiation.cc:407] Connection header received
0929 01:05:57.915062 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:57.915069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:57.915191 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:57.915267 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:57.920001 (+ 4734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.921250 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.925092 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:57.925864 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:57.926000 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:57.939828 (+ 13828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:57.939847 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:57.939867 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:57.939937 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:57.944746 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:57.946027 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:57.946040 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:57.946044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:57.946241 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:57.946917 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:57.946925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:57.946935 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:57.947588 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:57.948034 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:57.948790 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 01:05:57.949152 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:05:57.950889 22708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:58.054891 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22585
I20260929 01:05:58.075898 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22719
W20260929 01:05:58.075969 22508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.21.193:36923 (error 108)
I20260929 01:05:58.077950 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.077299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52610)
0929 01:05:58.077563 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.077722 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 01:05:58.078305 22508 catalog_manager.cc:4584] TS 11726aa186404cdcbe799f55877455ee (127.3.21.193:36923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ef332454558401aa69d64c8c710e0f6: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Sep 29 01:05:58 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.194@KRBTEST.COM
I20260929 01:05:58.095813 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.076845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.3.21.194:34265)
0929 01:05:58.077079 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:58.077087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:58.077239 (+ 152us) server_negotiation.cc:407] Connection header received
0929 01:05:58.077554 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:58.077558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:58.077646 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:58.077731 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:58.079527 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:58.080312 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:58.081774 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:58.082418 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:58.082814 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:58.088103 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:58.088124 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:58.088150 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:58.088194 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:58.091683 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:58.093114 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:58.093124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:58.093127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:58.093317 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:58.093816 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:58.093820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:58.093822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:58.094636 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:58.094968 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:58.095240 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 01:05:58.095439 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:05:58.096597 22842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:58.132812 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.131861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52612)
0929 01:05:58.132319 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.132532 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 01:05:58.135267 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.134442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52616)
0929 01:05:58.134883 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.135029 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260929 01:05:58.196398 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.195831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52620)
0929 01:05:58.196084 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.196192 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 01:05:58.198346 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.197664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52632)
0929 01:05:58.197949 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.198052 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 01:05:58.199245 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22719
I20260929 01:05:58.217507 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22853
Sep 29 01:05:58 dist-test-slave-k0b7 krb5kdc[22479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.195@KRBTEST.COM
I20260929 01:05:58.236701 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.218688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.3.21.195:34303)
0929 01:05:58.218856 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:58.218861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:58.218880 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:05:58.218938 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:58.218942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:58.219022 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:58.219133 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:58.220712 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:58.221498 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:58.223182 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:58.223624 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:58.223819 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:58.229183 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:58.229202 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:58.229221 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:58.229272 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:58.232721 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:58.233882 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:58.233891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:58.233893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:58.234057 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:58.234914 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:58.234920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:58.234922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:58.235454 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:58.235734 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:58.236131 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 01:05:58.236338 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:05:58.237505 22976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:58.279784 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.279262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52648)
0929 01:05:58.279514 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.279608 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 01:05:58.281406 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.280812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:36923 (local address 127.0.0.1:52664)
0929 01:05:58.281068 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.281260 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:36923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 01:05:58.338572 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22853
I20260929 01:05:58.356675 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 22492
I20260929 01:05:58.357957 22554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:05:58.480382 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 22492
2026-09-29T01:05:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4864 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T01:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:58Z Disabled control of system clock
I20260929 01:05:58.564222 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37381
--webserver_interface=127.3.21.254
--webserver_port=0
--builtin_ntp_servers=127.3.21.212:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:05:58.932291 23039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:58.932647 23039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:58.932737 23039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:58.940740 23039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:58.940871 23039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:58.940909 23039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:58.940960 23039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:58.941015 23039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:58.951287 23039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:58.952620 23039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:58.954469 23039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:58.967918 23044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:58.968056 23045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:58.968585 23047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:58.969007 23039 server_base.cc:1034] running on GCE node
I20260929 01:05:58.969808 23039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:58.971436 23039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:58.972662 23039 hybrid_clock.cc:648] HybridClock initialized: now 1790643958972604 us; error 48 us; skew 500 ppm
I20260929 01:05:58.973061 23039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:58.984360 23039 webserver.cc:460] Webserver started at http://127.3.21.254:38545/ using document root <none> and password file <none>
I20260929 01:05:58.985183 23039 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:58.985289 23039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:58.985684 23039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:58.989034 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "7c40f347df284fbaa290de96af96e162"
format_stamp: "Formatted at 2026-09-29 01:05:58 on dist-test-slave-k0b7"
server_key: "e8c1e664eaa3e28442a4938e0085e7a9"
server_key_iv: "a895a547f194622813dea84b8d3c0de3"
server_key_version: "encryptionkey@0"
I20260929 01:05:58.990020 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "7c40f347df284fbaa290de96af96e162"
format_stamp: "Formatted at 2026-09-29 01:05:58 on dist-test-slave-k0b7"
server_key: "e8c1e664eaa3e28442a4938e0085e7a9"
server_key_iv: "a895a547f194622813dea84b8d3c0de3"
server_key_version: "encryptionkey@0"
I20260929 01:05:58.995940 23039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 01:05:59.000733 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:59.002460 23039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 01:05:59.002767 23039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "7c40f347df284fbaa290de96af96e162"
format_stamp: "Formatted at 2026-09-29 01:05:58 on dist-test-slave-k0b7"
server_key: "e8c1e664eaa3e28442a4938e0085e7a9"
server_key_iv: "a895a547f194622813dea84b8d3c0de3"
server_key_version: "encryptionkey@0"
I20260929 01:05:59.003015 23039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:59.020574 23039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:59.021663 23039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:59.022035 23039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:59.044466 23039 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:37381
I20260929 01:05:59.044476 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:37381 every 8 connection(s)
I20260929 01:05:59.046128 23039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:59.050781 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:59.052947 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23039
I20260929 01:05:59.053148 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:05:59.053682 3159 external_mini_cluster.cc:1428] Setting key c2ebcc4ec089c8ae688eb9a42aafcd83
I20260929 01:05:59.064261 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162: Bootstrap starting.
I20260929 01:05:59.067855 23105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:59.069015 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:59.056058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.3.21.254:37381)
0929 01:05:59.057258 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:59.057275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:59.057325 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:05:59.059428 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:59.059454 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:59.059971 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:59.060430 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:59.061103 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:59.062647 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:59.064061 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:59.064889 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:59.065099 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:59.066431 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:59.066439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:59.066454 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:59.067112 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:59.067211 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:59.067235 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:59.067370 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:59.067611 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 01:05:59.067981 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":404,"threads_started":1}
I20260929 01:05:59.069059 23105 log.cc:824] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:59.072291 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162: No bootstrap required, opened a new log
I20260929 01:05:59.082998 23105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c40f347df284fbaa290de96af96e162" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37381 } }
I20260929 01:05:59.083629 23105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:59.083765 23105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c40f347df284fbaa290de96af96e162, State: Initialized, Role: FOLLOWER
I20260929 01:05:59.084424 23105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c40f347df284fbaa290de96af96e162" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37381 } }
I20260929 01:05:59.084656 23105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:59.084761 23105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:59.084897 23105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:59.087291 23105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c40f347df284fbaa290de96af96e162" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37381 } }
I20260929 01:05:59.087894 23105 leader_election.cc:302] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c40f347df284fbaa290de96af96e162; no voters:
I20260929 01:05:59.088672 23105 leader_election.cc:288] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:59.088886 23110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:59.089759 23110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [term 1 LEADER]: Becoming Leader. State: Replica: 7c40f347df284fbaa290de96af96e162, State: Running, Role: LEADER
I20260929 01:05:59.090353 23110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c40f347df284fbaa290de96af96e162" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37381 } }
I20260929 01:05:59.091223 23105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:59.093555 23112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c40f347df284fbaa290de96af96e162. Latest consensus state: current_term: 1 leader_uuid: "7c40f347df284fbaa290de96af96e162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c40f347df284fbaa290de96af96e162" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37381 } } }
I20260929 01:05:59.094115 23112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:59.094763 23111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c40f347df284fbaa290de96af96e162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c40f347df284fbaa290de96af96e162" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37381 } } }
I20260929 01:05:59.095153 23111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162 [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:59.099071 23118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:59.103662 23118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:59.114125 23118 catalog_manager.cc:1347] Generated new cluster ID: 347be66ea4d94237a4215e7fd6d35bb2
I20260929 01:05:59.114300 23118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:59.135509 23118 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:59.137055 23118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:59.151116 23118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162: Generated new TSK 0
I20260929 01:05:59.152320 23118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:05:59.161290 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.193:0
--local_ip_for_outbound_sockets=127.3.21.193
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37381
--builtin_ntp_servers=127.3.21.212:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:05:59.532899 23129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:59.533232 23129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:59.533332 23129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:59.541061 23129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:59.541203 23129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:59.541316 23129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.193
I20260929 01:05:59.551419 23129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.21.193
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.193
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:05:59.553120 23129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:59.555253 23129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:59.569721 23135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:59.569712 23137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:59.570142 23134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:59.571158 23129 server_base.cc:1034] running on GCE node
I20260929 01:05:59.571904 23129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:59.573072 23129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:59.574325 23129 hybrid_clock.cc:648] HybridClock initialized: now 1790643959574251 us; error 68 us; skew 500 ppm
I20260929 01:05:59.574853 23129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:05:59.577788 23129 webserver.cc:460] Webserver started at http://127.3.21.193:39249/ using document root <none> and password file <none>
I20260929 01:05:59.578651 23129 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:59.578774 23129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:59.579152 23129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:59.582352 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data/instance:
uuid: "87b6b0b47d1344dca3173b79fc51e425"
format_stamp: "Formatted at 2026-09-29 01:05:59 on dist-test-slave-k0b7"
server_key: "f5a54927c011e93edac35903c85d9ab9"
server_key_iv: "738f325e8f33caead85cfc80d562a387"
server_key_version: "encryptionkey@0"
I20260929 01:05:59.583323 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance:
uuid: "87b6b0b47d1344dca3173b79fc51e425"
format_stamp: "Formatted at 2026-09-29 01:05:59 on dist-test-slave-k0b7"
server_key: "f5a54927c011e93edac35903c85d9ab9"
server_key_iv: "738f325e8f33caead85cfc80d562a387"
server_key_version: "encryptionkey@0"
I20260929 01:05:59.589094 23129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 01:05:59.593199 23143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:59.595259 23129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:05:59.595479 23129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal
uuid: "87b6b0b47d1344dca3173b79fc51e425"
format_stamp: "Formatted at 2026-09-29 01:05:59 on dist-test-slave-k0b7"
server_key: "f5a54927c011e93edac35903c85d9ab9"
server_key_iv: "738f325e8f33caead85cfc80d562a387"
server_key_version: "encryptionkey@0"
I20260929 01:05:59.595791 23129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:59.618373 23129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:59.619378 23129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:59.619793 23129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:59.620873 23129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:59.622539 23129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:59.622668 23129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:59.622802 23129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:59.622889 23129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:59.660408 23129 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.193:41313
I20260929 01:05:59.660506 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.193:41313 every 8 connection(s)
I20260929 01:05:59.661821 23129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:59.671007 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23129
I20260929 01:05:59.671198 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:59.671701 3159 external_mini_cluster.cc:1428] Setting key df8f630dea3bc314f0e97329e277b093
I20260929 01:05:59.675066 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.194:0
--local_ip_for_outbound_sockets=127.3.21.194
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37381
--builtin_ntp_servers=127.3.21.212:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:05:59.677719 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:59.664905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:40063 (local address 127.3.21.254:37381)
0929 01:05:59.665542 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:59.665548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:59.666987 (+ 1439us) server_negotiation.cc:407] Connection header received
0929 01:05:59.669153 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:59.669161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:59.669279 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:59.669425 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:59.670771 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:59.671690 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:59.674100 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:59.674653 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:59.674771 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:59.676732 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:59.676739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:59.676743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:05:59.676764 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:59.676806 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:05:59.676819 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:59.676903 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:59.677244 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 01:05:59.677353 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":375,"threads_started":1}
I20260929 01:05:59.679349 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:59.665610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37381 (local address 127.3.21.193:40063)
0929 01:05:59.666794 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:59.666837 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:59.668806 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:59.669704 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:59.669713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:59.669814 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:59.670466 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:59.670579 (+ 113us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:59.671920 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:59.671927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:59.673814 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:59.673832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:59.674924 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:59.674932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:59.675311 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:59.676168 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:05:59.676175 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:59.676240 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:05:59.676377 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:59.676942 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:59.676957 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:59.676971 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 01:05:59.677200 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 01:05:59.677736 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":381,"threads_started":1}
I20260929 01:05:59.681516 23256 heartbeater.cc:347] Connected to a master server at 127.3.21.254:37381
I20260929 01:05:59.681916 23256 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:59.683053 23256 heartbeater.cc:511] Master 127.3.21.254:37381 requested a full tablet report, sending...
I20260929 01:05:59.686151 23070 ts_manager.cc:194] Registered new tserver with Master: 87b6b0b47d1344dca3173b79fc51e425 (127.3.21.193:41313)
I20260929 01:05:59.687897 23070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.193:40063
W20260929 01:06:00.077826 23260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:00.078168 23260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:00.078332 23260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:00.087191 23260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:00.087304 23260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:00.087476 23260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.194
I20260929 01:06:00.101296 23260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.21.194
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.194
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:00.102789 23260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:00.104645 23260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:00.120932 23265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:00.120836 23268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:00.120875 23266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:00.122215 23260 server_base.cc:1034] running on GCE node
I20260929 01:06:00.123013 23260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:00.124121 23260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:00.125463 23260 hybrid_clock.cc:648] HybridClock initialized: now 1790643960125371 us; error 72 us; skew 500 ppm
I20260929 01:06:00.125851 23260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:06:00.129611 23260 webserver.cc:460] Webserver started at http://127.3.21.194:46619/ using document root <none> and password file <none>
I20260929 01:06:00.131021 23260 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:00.131407 23260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:00.132153 23260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:00.136312 23260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data/instance:
uuid: "d58f76fddab5438fbf873f62bc542503"
format_stamp: "Formatted at 2026-09-29 01:06:00 on dist-test-slave-k0b7"
server_key: "10365a1f6a847e40304bfc387b1f0a83"
server_key_iv: "fe8c13e35c896de6a2568253ca804971"
server_key_version: "encryptionkey@0"
I20260929 01:06:00.137310 23260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance:
uuid: "d58f76fddab5438fbf873f62bc542503"
format_stamp: "Formatted at 2026-09-29 01:06:00 on dist-test-slave-k0b7"
server_key: "10365a1f6a847e40304bfc387b1f0a83"
server_key_iv: "fe8c13e35c896de6a2568253ca804971"
server_key_version: "encryptionkey@0"
I20260929 01:06:00.143975 23260 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:06:00.148480 23274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:00.150864 23260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260929 01:06:00.151234 23260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal
uuid: "d58f76fddab5438fbf873f62bc542503"
format_stamp: "Formatted at 2026-09-29 01:06:00 on dist-test-slave-k0b7"
server_key: "10365a1f6a847e40304bfc387b1f0a83"
server_key_iv: "fe8c13e35c896de6a2568253ca804971"
server_key_version: "encryptionkey@0"
I20260929 01:06:00.151543 23260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:00.212029 23260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:00.213047 23260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:00.213414 23260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:00.214885 23260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:00.216827 23260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:00.216989 23260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:00.217140 23260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:00.217244 23260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:00.257791 23260 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.194:42803
I20260929 01:06:00.257836 23386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.194:42803 every 8 connection(s)
I20260929 01:06:00.259299 23260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:00.261008 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23260
I20260929 01:06:00.261217 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:00.261765 3159 external_mini_cluster.cc:1428] Setting key 3a1c703540ae546a1a61d612513520a9
I20260929 01:06:00.265923 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.21.195:0
--local_ip_for_outbound_sockets=127.3.21.195
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37381
--builtin_ntp_servers=127.3.21.212:35939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:00.277423 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.263146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.194:41291 (local address 127.3.21.254:37381)
0929 01:06:00.263647 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:00.263652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:00.265762 (+ 2110us) server_negotiation.cc:407] Connection header received
0929 01:06:00.268835 (+ 3073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:00.268840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:00.268926 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:00.269107 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:00.270670 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.271426 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.274120 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.274643 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.274804 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.276477 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:00.276482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:00.276485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:00.276507 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:00.276554 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:00.276568 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:00.276680 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:00.276911 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 01:06:00.277029 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260929 01:06:00.278638 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.264111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37381 (local address 127.3.21.194:41291)
0929 01:06:00.265566 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.265617 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:00.268433 (+ 2816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:00.269384 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:00.269399 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:00.269538 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:00.270383 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.270411 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.271619 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.271628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.273890 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.273912 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.274945 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.274954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.275293 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.276079 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:00.276082 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:00.276126 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:00.276228 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:00.276673 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:00.276683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:00.276694 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 01:06:00.276899 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 01:06:00.277392 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":455,"threads_started":1}
I20260929 01:06:00.280710 23387 heartbeater.cc:347] Connected to a master server at 127.3.21.254:37381
I20260929 01:06:00.281113 23387 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:00.282070 23387 heartbeater.cc:511] Master 127.3.21.254:37381 requested a full tablet report, sending...
I20260929 01:06:00.283963 23070 ts_manager.cc:194] Registered new tserver with Master: d58f76fddab5438fbf873f62bc542503 (127.3.21.194:42803)
I20260929 01:06:00.285336 23070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.194:41291
W20260929 01:06:00.629585 23391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:00.629908 23391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:00.629998 23391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:00.638809 23391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:00.638957 23391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:00.639087 23391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.21.195
I20260929 01:06:00.649112 23391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.21.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.21.195
--webserver_port=0
--tserver_master_addrs=127.3.21.254:37381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.21.195
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:00.650473 23391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:00.652464 23391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:00.668346 23396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:00.668315 23397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:00.668915 23399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:00.669476 23391 server_base.cc:1034] running on GCE node
I20260929 01:06:00.670193 23391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:00.671391 23391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:00.672645 23391 hybrid_clock.cc:648] HybridClock initialized: now 1790643960672603 us; error 42 us; skew 500 ppm
I20260929 01:06:00.673089 23391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:06:00.675647 23391 webserver.cc:460] Webserver started at http://127.3.21.195:34419/ using document root <none> and password file <none>
I20260929 01:06:00.676386 23391 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:00.676524 23391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:00.676857 23391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:00.679894 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data/instance:
uuid: "58173d1c0d5f4cc38b426cc0e5ff5010"
format_stamp: "Formatted at 2026-09-29 01:06:00 on dist-test-slave-k0b7"
server_key: "0616ecaa35de81cbbbde2852cc264f47"
server_key_iv: "d4ed91ee6ec348ba30256e04843096ec"
server_key_version: "encryptionkey@0"
I20260929 01:06:00.680716 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance:
uuid: "58173d1c0d5f4cc38b426cc0e5ff5010"
format_stamp: "Formatted at 2026-09-29 01:06:00 on dist-test-slave-k0b7"
server_key: "0616ecaa35de81cbbbde2852cc264f47"
server_key_iv: "d4ed91ee6ec348ba30256e04843096ec"
server_key_version: "encryptionkey@0"
I20260929 01:06:00.686313 23391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:06:00.690325 23405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:00.692041 23391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:00.692245 23391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal
uuid: "58173d1c0d5f4cc38b426cc0e5ff5010"
format_stamp: "Formatted at 2026-09-29 01:06:00 on dist-test-slave-k0b7"
server_key: "0616ecaa35de81cbbbde2852cc264f47"
server_key_iv: "d4ed91ee6ec348ba30256e04843096ec"
server_key_version: "encryptionkey@0"
I20260929 01:06:00.692490 23391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:00.692917 23256 heartbeater.cc:503] Master 127.3.21.254:37381 was elected leader, sending a full tablet report...
I20260929 01:06:00.712302 23391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:00.713187 23391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:00.713513 23391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:00.714588 23391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:00.716142 23391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:00.716295 23391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:00.716424 23391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:00.716513 23391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:00.756131 23391 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.195:40617
I20260929 01:06:00.756302 23517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.195:40617 every 8 connection(s)
I20260929 01:06:00.757741 23391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:00.759255 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23391
I20260929 01:06:00.759428 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:00.759949 3159 external_mini_cluster.cc:1428] Setting key 2c3cc6801ff4abe191f40278e60c656d
I20260929 01:06:00.774220 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.761044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.195:59035 (local address 127.3.21.254:37381)
0929 01:06:00.761312 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:00.761318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:00.763633 (+ 2315us) server_negotiation.cc:407] Connection header received
0929 01:06:00.765970 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:00.765975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:00.766061 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:00.766185 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:00.767843 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.768848 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.771162 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.771576 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.771729 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.773324 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:00.773329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:00.773333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:00.773355 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:00.773389 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:00.773402 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:00.773506 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:00.773780 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 01:06:00.773893 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260929 01:06:00.775436 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.761642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37381 (local address 127.3.21.195:59035)
0929 01:06:00.763465 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.763522 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:00.765351 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:00.766569 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:00.766600 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:00.766767 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:00.767637 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.767658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.769089 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.769097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.770984 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.770999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.771839 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.771847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.772105 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.772836 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:00.772839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:00.772893 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:00.772989 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:00.773507 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:00.773520 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:00.773540 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:06:00.773757 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 01:06:00.774223 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":1002,"threads_started":1}
I20260929 01:06:00.777168 23518 heartbeater.cc:347] Connected to a master server at 127.3.21.254:37381
I20260929 01:06:00.777516 23518 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:00.778357 23518 heartbeater.cc:511] Master 127.3.21.254:37381 requested a full tablet report, sending...
I20260929 01:06:00.779915 23070 ts_manager.cc:194] Registered new tserver with Master: 58173d1c0d5f4cc38b426cc0e5ff5010 (127.3.21.195:40617)
I20260929 01:06:00.780812 23070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.21.195:59035
I20260929 01:06:00.792208 3159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:00.803768 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.797197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.3.21.254:37381)
0929 01:06:00.797459 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:00.797468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:00.797748 (+ 280us) server_negotiation.cc:407] Connection header received
0929 01:06:00.797972 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:00.797978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:00.798056 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:00.798179 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:00.798776 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.799609 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.801107 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.801678 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.801822 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.802982 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:00.802988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:00.802993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:00.803023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:00.803055 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:00.803068 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:00.803179 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:00.803314 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 01:06:00.803414 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 01:06:00.808157 23070 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:00.815474 23070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:00.854818 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.829828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:42803 (local address 127.0.0.1:36732)
0929 01:06:00.843306 (+ 13478us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.843364 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:00.843620 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:00.846102 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:00.846111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:00.846143 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:00.846641 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.846659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.848628 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.848636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.851504 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.851527 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.853398 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.853405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.853748 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.853821 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 01:06:00.853963 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:06:00.854373 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13151,"mutex_wait_us":148,"thread_start_us":14925,"threads_started":1}
I20260929 01:06:00.855019 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.831666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53490 (local address 127.3.21.193:41313)
0929 01:06:00.844706 (+ 13040us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:00.844719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:00.845725 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 01:06:00.845932 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:00.845941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:00.846159 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:00.846392 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:00.847339 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.849041 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.852260 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.853692 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.853860 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.854116 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:00.854357 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 01:06:00.854631 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12840,"thread_start_us":12665,"threads_started":1}
I20260929 01:06:00.855670 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.830334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.3.21.194:42803)
0929 01:06:00.845303 (+ 14969us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:00.845315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:00.845350 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:06:00.845469 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:00.845479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:00.845815 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:00.846903 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:00.847206 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.848461 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.851766 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.853168 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.853336 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.853786 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:00.855157 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0929 01:06:00.855342 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14786,"thread_start_us":14752,"threads_started":1}
I20260929 01:06:00.855784 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.831266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53490)
0929 01:06:00.845578 (+ 14312us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.845596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:00.845743 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:00.846748 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:00.846752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:00.846773 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:00.847157 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.847166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.849230 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.849236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.852044 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.852055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.853960 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.853967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.854127 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.854158 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:06:00.855282 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0929 01:06:00.855416 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14194,"mutex_wait_us":118,"thread_start_us":13816,"threads_started":1}
I20260929 01:06:00.855791 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.830116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.3.21.195:40617)
0929 01:06:00.842559 (+ 12443us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:00.842574 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:00.845361 (+ 2787us) server_negotiation.cc:407] Connection header received
0929 01:06:00.846173 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:00.846183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:00.846413 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:00.846639 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:00.847833 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.849594 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.852650 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.854149 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.854317 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.854609 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:00.855180 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 01:06:00.855372 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12128,"thread_start_us":13771,"threads_started":1}
I20260929 01:06:00.856508 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.829947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:40617 (local address 127.0.0.1:41654)
0929 01:06:00.845211 (+ 15264us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.845234 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:00.845942 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:00.847024 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:00.847030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:00.847065 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:00.847527 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.847541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.849760 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.849766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.852410 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:00.852427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:00.854776 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:00.854784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:00.854972 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:00.855017 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:06:00.855969 (+ 952us) client_negotiation.cc:240] Negotiation successful
0929 01:06:00.856163 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15076,"mutex_wait_us":5,"thread_start_us":12837,"threads_started":1}
I20260929 01:06:00.859853 23191 tablet_service.cc:1467] Processing CreateTablet for tablet 808bc694f02b4007be5b7b2e5abe38cb (DEFAULT_TABLE table=test-table [id=294d1a19b23d4a62af1956e5f398d477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:00.859964 23322 tablet_service.cc:1467] Processing CreateTablet for tablet 808bc694f02b4007be5b7b2e5abe38cb (DEFAULT_TABLE table=test-table [id=294d1a19b23d4a62af1956e5f398d477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:00.859964 23453 tablet_service.cc:1467] Processing CreateTablet for tablet 808bc694f02b4007be5b7b2e5abe38cb (DEFAULT_TABLE table=test-table [id=294d1a19b23d4a62af1956e5f398d477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:00.861372 23191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808bc694f02b4007be5b7b2e5abe38cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:00.861382 23453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808bc694f02b4007be5b7b2e5abe38cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:00.861382 23322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808bc694f02b4007be5b7b2e5abe38cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:00.874197 23536 tablet_bootstrap.cc:492] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Bootstrap starting.
I20260929 01:06:00.878326 23536 tablet_bootstrap.cc:654] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:00.879452 23536 log.cc:824] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:00.882251 23538 tablet_bootstrap.cc:492] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Bootstrap starting.
I20260929 01:06:00.882654 23537 tablet_bootstrap.cc:492] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Bootstrap starting.
I20260929 01:06:00.883251 23536 tablet_bootstrap.cc:492] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: No bootstrap required, opened a new log
I20260929 01:06:00.883543 23536 ts_tablet_manager.cc:1397] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 01:06:00.886345 23538 tablet_bootstrap.cc:654] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:00.886345 23537 tablet_bootstrap.cc:654] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:00.887816 23538 log.cc:824] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:00.887816 23537 log.cc:824] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:00.891875 23538 tablet_bootstrap.cc:492] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: No bootstrap required, opened a new log
I20260929 01:06:00.892369 23538 ts_tablet_manager.cc:1397] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 01:06:00.892529 23537 tablet_bootstrap.cc:492] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: No bootstrap required, opened a new log
I20260929 01:06:00.892828 23537 ts_tablet_manager.cc:1397] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 01:06:00.893471 23536 raft_consensus.cc:348] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:00.893992 23536 raft_consensus.cc:374] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:00.894102 23536 raft_consensus.cc:729] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b6b0b47d1344dca3173b79fc51e425, State: Initialized, Role: FOLLOWER
I20260929 01:06:00.894872 23536 consensus_queue.cc:260] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:00.898567 23536 ts_tablet_manager.cc:1428] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 01:06:00.904502 23538 raft_consensus.cc:348] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:00.904491 23537 raft_consensus.cc:348] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:00.904958 23538 raft_consensus.cc:374] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:00.904976 23537 raft_consensus.cc:374] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:00.905087 23538 raft_consensus.cc:729] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58173d1c0d5f4cc38b426cc0e5ff5010, State: Initialized, Role: FOLLOWER
I20260929 01:06:00.905107 23537 raft_consensus.cc:729] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d58f76fddab5438fbf873f62bc542503, State: Initialized, Role: FOLLOWER
I20260929 01:06:00.905793 23537 consensus_queue.cc:260] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:00.905794 23538 consensus_queue.cc:260] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:00.907126 23518 heartbeater.cc:503] Master 127.3.21.254:37381 was elected leader, sending a full tablet report...
I20260929 01:06:00.907226 23387 heartbeater.cc:503] Master 127.3.21.254:37381 was elected leader, sending a full tablet report...
I20260929 01:06:00.907943 23537 ts_tablet_manager.cc:1428] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 01:06:00.907955 23538 ts_tablet_manager.cc:1428] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260929 01:06:00.914830 23257 tablet.cc:2369] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:06:01.009799 23519 tablet.cc:2369] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:06:01.012424 23388 tablet.cc:2369] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:01.107352 23542 raft_consensus.cc:482] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:01.107754 23542 raft_consensus.cc:504] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:01.109723 23542 leader_election.cc:288] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58173d1c0d5f4cc38b426cc0e5ff5010 (127.3.21.195:40617), d58f76fddab5438fbf873f62bc542503 (127.3.21.194:42803)
I20260929 01:06:01.116827 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.110402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:40945 (local address 127.3.21.194:42803)
0929 01:06:01.110635 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:01.110643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:01.111298 (+ 655us) server_negotiation.cc:407] Connection header received
0929 01:06:01.111558 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:01.111573 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:01.111645 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:01.111744 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:01.112349 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.113081 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.115076 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.115867 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.116047 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.116122 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:01.116396 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 01:06:01.116527 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:06:01.116925 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.110154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.194:42803 (local address 127.3.21.193:40945)
0929 01:06:01.111182 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.111198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:01.111388 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:01.111780 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:01.111785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:01.111875 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:01.112202 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:01.112212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.113218 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.113224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:01.114858 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:01.114875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.116056 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.116072 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:01.116229 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.116255 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:06:01.116364 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:06:01.116509 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"spinlock_wait_cycles":41216,"thread_start_us":726,"threads_started":1}
I20260929 01:06:01.117573 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.110845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.195:40617 (local address 127.3.21.193:42371)
0929 01:06:01.111660 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.111675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:01.111854 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:01.112222 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:01.112226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:01.112250 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:01.112566 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:01.112575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.113630 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.113636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:01.115360 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:01.115372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.116593 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.116599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:01.116761 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.116784 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:06:01.116916 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 01:06:01.117265 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":468,"threads_started":1}
I20260929 01:06:01.117726 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.110308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.21.193:42371 (local address 127.3.21.195:40617)
0929 01:06:01.110577 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:01.110583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:01.111791 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 01:06:01.112010 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:01.112016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:01.112091 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:01.112196 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:01.112751 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.113469 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.115544 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.116372 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.116566 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.116645 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:01.117312 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 01:06:01.117458 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:06:01.117959 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808bc694f02b4007be5b7b2e5abe38cb" candidate_uuid: "87b6b0b47d1344dca3173b79fc51e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58f76fddab5438fbf873f62bc542503" is_pre_election: true
I20260929 01:06:01.118784 23342 raft_consensus.cc:2454] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b6b0b47d1344dca3173b79fc51e425 in term 0.
I20260929 01:06:01.118764 23473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808bc694f02b4007be5b7b2e5abe38cb" candidate_uuid: "87b6b0b47d1344dca3173b79fc51e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" is_pre_election: true
I20260929 01:06:01.119338 23473 raft_consensus.cc:2454] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b6b0b47d1344dca3173b79fc51e425 in term 0.
I20260929 01:06:01.119633 23147 leader_election.cc:302] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b6b0b47d1344dca3173b79fc51e425, d58f76fddab5438fbf873f62bc542503; no voters:
I20260929 01:06:01.120282 23542 raft_consensus.cc:2781] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:01.120484 23542 raft_consensus.cc:482] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:01.120646 23542 raft_consensus.cc:3037] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:01.123322 23542 raft_consensus.cc:504] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:01.124269 23542 leader_election.cc:288] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [CANDIDATE]: Term 1 election: Requested vote from peers 58173d1c0d5f4cc38b426cc0e5ff5010 (127.3.21.195:40617), d58f76fddab5438fbf873f62bc542503 (127.3.21.194:42803)
I20260929 01:06:01.124819 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808bc694f02b4007be5b7b2e5abe38cb" candidate_uuid: "87b6b0b47d1344dca3173b79fc51e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d58f76fddab5438fbf873f62bc542503"
I20260929 01:06:01.124840 23473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808bc694f02b4007be5b7b2e5abe38cb" candidate_uuid: "87b6b0b47d1344dca3173b79fc51e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010"
I20260929 01:06:01.125061 23342 raft_consensus.cc:3037] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:01.125075 23473 raft_consensus.cc:3037] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:01.127576 23473 raft_consensus.cc:2454] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b6b0b47d1344dca3173b79fc51e425 in term 1.
I20260929 01:06:01.127787 23342 raft_consensus.cc:2454] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b6b0b47d1344dca3173b79fc51e425 in term 1.
I20260929 01:06:01.128214 23145 leader_election.cc:302] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58173d1c0d5f4cc38b426cc0e5ff5010, 87b6b0b47d1344dca3173b79fc51e425; no voters:
I20260929 01:06:01.128624 23542 raft_consensus.cc:2781] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:01.129132 23542 raft_consensus.cc:686] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 1 LEADER]: Becoming Leader. State: Replica: 87b6b0b47d1344dca3173b79fc51e425, State: Running, Role: LEADER
I20260929 01:06:01.129655 23542 consensus_queue.cc:237] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } }
I20260929 01:06:01.135046 23070 catalog_manager.cc:5495] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87b6b0b47d1344dca3173b79fc51e425 (127.3.21.193). New cstate: current_term: 1 leader_uuid: "87b6b0b47d1344dca3173b79fc51e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87b6b0b47d1344dca3173b79fc51e425" member_type: VOTER last_known_addr { host: "127.3.21.193" port: 41313 } health_report { overall_health: HEALTHY } } }
I20260929 01:06:01.158705 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.153003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.3.21.193:41313)
0929 01:06:01.153266 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:01.153277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:01.153293 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:01.153386 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:01.153389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:01.153487 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:01.153649 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:01.154028 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.154879 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.156298 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.156753 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.156905 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.157467 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:01.158057 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:01.158246 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:01.158299 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 01:06:01.158387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:06:01.168017 23342 raft_consensus.cc:1261] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 1 FOLLOWER]: Refusing update from remote peer 87b6b0b47d1344dca3173b79fc51e425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:01.168028 23473 raft_consensus.cc:1261] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 1 FOLLOWER]: Refusing update from remote peer 87b6b0b47d1344dca3173b79fc51e425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:01.169430 23547 consensus_queue.cc:1035] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58173d1c0d5f4cc38b426cc0e5ff5010" member_type: VOTER last_known_addr { host: "127.3.21.195" port: 40617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:01.169859 23542 consensus_queue.cc:1035] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d58f76fddab5438fbf873f62bc542503" member_type: VOTER last_known_addr { host: "127.3.21.194" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:01.190284 23551 mvcc.cc:205] Tried to move back new op lower bound from 7334477664932491264 to 7334477664791371776. Current Snapshot: MvccSnapshot[applied={T|T < 7334477664932491264}]
I20260929 01:06:01.201535 23069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:01.201845 23069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:01.205866 23069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c40f347df284fbaa290de96af96e162: Sending DeleteTablet for 3 replicas of tablet 808bc694f02b4007be5b7b2e5abe38cb
I20260929 01:06:01.207212 23322 tablet_service.cc:1514] Processing DeleteTablet for tablet 808bc694f02b4007be5b7b2e5abe38cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:01 UTC) from {username='slave'} at 127.0.0.1:36732
I20260929 01:06:01.207177 23453 tablet_service.cc:1514] Processing DeleteTablet for tablet 808bc694f02b4007be5b7b2e5abe38cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:01 UTC) from {username='slave'} at 127.0.0.1:41654
I20260929 01:06:01.207975 23191 tablet_service.cc:1514] Processing DeleteTablet for tablet 808bc694f02b4007be5b7b2e5abe38cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:01 UTC) from {username='slave'} at 127.0.0.1:53490
I20260929 01:06:01.208643 23561 tablet_replica.cc:330] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: stopping tablet replica
I20260929 01:06:01.208946 23560 tablet_replica.cc:330] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: stopping tablet replica
I20260929 01:06:01.209550 23562 tablet_replica.cc:330] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: stopping tablet replica
I20260929 01:06:01.209944 23562 raft_consensus.cc:2229] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:06:01.209986 23560 raft_consensus.cc:2229] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:01.210358 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23129
I20260929 01:06:01.209522 23561 raft_consensus.cc:2229] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:01.210724 23562 raft_consensus.cc:2258] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:01.211086 23560 raft_consensus.cc:2258] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:01.211659 23561 raft_consensus.cc:2258] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:01.214334 23562 ts_tablet_manager.cc:1905] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:01.215739 23560 ts_tablet_manager.cc:1905] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:01.221000 23561 ts_tablet_manager.cc:1905] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:01.221359 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.212569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.3.21.193:41313)
0929 01:06:01.212771 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:01.212777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:01.212798 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:06:01.212866 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:01.212870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:01.212962 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:01.213077 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:01.213762 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.214919 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.216973 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.217666 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.217768 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.219432 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:01.219446 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:01.219465 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:01.220344 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:01.220419 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:01.220450 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:01.220545 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:01.220823 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 01:06:01.220942 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:01.222868 23251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:01.222519 23562 ts_tablet_manager.cc:1918] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:01.226550 23562 log.cc:1196] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-0/wal/wals/808bc694f02b4007be5b7b2e5abe38cb
I20260929 01:06:01.227905 23560 ts_tablet_manager.cc:1918] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:01.228088 23560 log.cc:1196] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-2/wal/wals/808bc694f02b4007be5b7b2e5abe38cb
I20260929 01:06:01.228633 23560 ts_tablet_manager.cc:1939] T 808bc694f02b4007be5b7b2e5abe38cb P 58173d1c0d5f4cc38b426cc0e5ff5010: Deleting consensus metadata
I20260929 01:06:01.228647 23561 ts_tablet_manager.cc:1918] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:01.228838 23561 log.cc:1196] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Deleting WAL directory at /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643611838815-3159-0/minicluster-data/ts-1/wal/wals/808bc694f02b4007be5b7b2e5abe38cb
I20260929 01:06:01.229365 23561 ts_tablet_manager.cc:1939] T 808bc694f02b4007be5b7b2e5abe38cb P d58f76fddab5438fbf873f62bc542503: Deleting consensus metadata
I20260929 01:06:01.230880 23055 catalog_manager.cc:4841] TS 58173d1c0d5f4cc38b426cc0e5ff5010 (127.3.21.195:40617): tablet 808bc694f02b4007be5b7b2e5abe38cb (table test-table [id=294d1a19b23d4a62af1956e5f398d477]) successfully deleted
I20260929 01:06:01.231087 23057 catalog_manager.cc:4841] TS d58f76fddab5438fbf873f62bc542503 (127.3.21.194:42803): tablet 808bc694f02b4007be5b7b2e5abe38cb (table test-table [id=294d1a19b23d4a62af1956e5f398d477]) successfully deleted
I20260929 01:06:01.328065 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23129
I20260929 01:06:01.328153 23562 ts_tablet_manager.cc:1939] T 808bc694f02b4007be5b7b2e5abe38cb P 87b6b0b47d1344dca3173b79fc51e425: Deleting consensus metadata
I20260929 01:06:01.349336 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23260
W20260929 01:06:01.349217 23054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.21.193:41313 (error 108)
I20260929 01:06:01.351461 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.350968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53508)
0929 01:06:01.351193 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.351313 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 01:06:01.352712 23054 catalog_manager.cc:4584] TS 87b6b0b47d1344dca3173b79fc51e425 (127.3.21.193:41313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 808bc694f02b4007be5b7b2e5abe38cb: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
I20260929 01:06:01.358103 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.350273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.3.21.194:42803)
0929 01:06:01.350457 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:01.350464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:01.350630 (+ 166us) server_negotiation.cc:407] Connection header received
0929 01:06:01.350881 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:01.350885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:01.350968 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:01.351197 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:01.351647 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.352441 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.354228 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.354815 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.354927 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.356574 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:01.356587 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:01.356650 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:01.357312 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:01.357364 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:01.357386 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:01.357489 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:01.357733 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 01:06:01.357838 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:06:01.358950 23382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:01.406464 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.405891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53514)
0929 01:06:01.406171 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.406292 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 01:06:01.408032 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.407403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53520)
0929 01:06:01.407757 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.407876 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260929 01:06:01.456483 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23260
I20260929 01:06:01.468845 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.468180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53532)
0929 01:06:01.468513 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.468619 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 01:06:01.470453 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.469896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53534)
0929 01:06:01.470206 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.470295 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260929 01:06:01.474607 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23391
I20260929 01:06:01.482739 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.475412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.3.21.195:40617)
0929 01:06:01.475688 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:01.475696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:01.475820 (+ 124us) server_negotiation.cc:407] Connection header received
0929 01:06:01.476063 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:01.476069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:01.476188 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:01.476317 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:01.476798 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.477601 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.479257 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:01.479699 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:01.479855 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:01.481132 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:01.481148 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:01.481164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:01.481762 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:01.481817 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:01.481839 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:01.481970 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:01.482129 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 01:06:01.482376 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:06:01.483537 23513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:01.551728 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.551030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53542)
0929 01:06:01.551390 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.551506 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260929 01:06:01.553066 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.552612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.193:41313 (local address 127.0.0.1:53546)
0929 01:06:01.552840 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.552927 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.193:41313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 01:06:01.577548 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23391
I20260929 01:06:01.594915 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23039
I20260929 01:06:01.596401 23100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:01.714229 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23039
2026-09-29T01:06:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3237 ms)
[----------] 6 tests from EncryptionPolicyTest (23965 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:01 dist-test-slave-k0b7 krb5kdc[23571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:01 dist-test-slave-k0b7 krb5kdc[23571](info): set up 2 sockets
Sep 29 01:06:01 dist-test-slave-k0b7 krb5kdc[23571](info): commencing operation
krb5kdc: starting...
W20260929 01:06:02.899441 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:03 dist-test-slave-k0b7 krb5kdc[23571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:03.201088 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:03.563956 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:03.564283 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:03.564452 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:03.572299 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:03.572438 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:03.572530 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:03.572625 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:03.572737 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:03.583333 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:03.585217 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:03.586839 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:03.600392 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:03.600382 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:03.600646 23592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:03.601325 23584 server_base.cc:1034] running on GCE node
I20260929 01:06:03.602252 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:03.603907 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:03.605141 23584 hybrid_clock.cc:648] HybridClock initialized: now 1790643963605093 us; error 48 us; skew 500 ppm
I20260929 01:06:03.605618 23584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:03 dist-test-slave-k0b7 krb5kdc[23571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:03.609027 23584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:03.610985 23584 webserver.cc:460] Webserver started at http://127.0.0.1:33271/ using document root <none> and password file <none>
I20260929 01:06:03.611719 23584 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:03.611825 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:03.612090 23584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:03.615458 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "f0a7ef3170b3439597cc7afc90c8482f"
format_stamp: "Formatted at 2026-09-29 01:06:03 on dist-test-slave-k0b7"
server_key: "528d38114fdda9c5858ef0f6c7cc224d"
server_key_iv: "1ee0e355e0e356262ed82bf8f63a54c0"
server_key_version: "encryptionkey@0"
I20260929 01:06:03.616257 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "f0a7ef3170b3439597cc7afc90c8482f"
format_stamp: "Formatted at 2026-09-29 01:06:03 on dist-test-slave-k0b7"
server_key: "528d38114fdda9c5858ef0f6c7cc224d"
server_key_iv: "1ee0e355e0e356262ed82bf8f63a54c0"
server_key_version: "encryptionkey@0"
I20260929 01:06:03.622139 23584 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:03.627015 23599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:03.629033 23584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:06:03.629289 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "f0a7ef3170b3439597cc7afc90c8482f"
format_stamp: "Formatted at 2026-09-29 01:06:03 on dist-test-slave-k0b7"
server_key: "528d38114fdda9c5858ef0f6c7cc224d"
server_key_iv: "1ee0e355e0e356262ed82bf8f63a54c0"
server_key_version: "encryptionkey@0"
I20260929 01:06:03.629535 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:03.646343 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:03.651659 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:03.652001 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:03.672747 23584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46075
I20260929 01:06:03.672942 23650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46075 every 8 connection(s)
I20260929 01:06:03.674302 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:03.678941 23651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:03.680751 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23584
I20260929 01:06:03.681008 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:03.681871 3159 external_mini_cluster.cc:1428] Setting key 78a7123b65f783efafa4dadcede60867
I20260929 01:06:03.695369 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f: Bootstrap starting.
Sep 29 01:06:03 dist-test-slave-k0b7 krb5kdc[23571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:03.699679 23651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:03.700776 23651 log.cc:824] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:03.704984 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f: No bootstrap required, opened a new log
I20260929 01:06:03.710794 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:03.684178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56583 (local address 127.0.0.1:46075)
0929 01:06:03.685059 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:03.685071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:03.685114 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:06:03.686850 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:03.686877 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:03.687359 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:03.687708 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:03.690046 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:03.691763 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:03.693737 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:03.694825 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:03.694969 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:03.700440 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:03.700449 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:03.700467 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:03.700518 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:03.704888 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:03.705930 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:03.705939 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:03.705947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:03.706189 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:03.706765 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:03.706770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:03.706773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:03.707776 (+ 1003us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:03.708140 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:03.708807 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 01:06:03.709211 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":652,"thread_start_us":411,"threads_started":1}
I20260929 01:06:03.715865 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:03.715910 23651 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a7ef3170b3439597cc7afc90c8482f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260929 01:06:03.716369 23651 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:03.716485 23651 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a7ef3170b3439597cc7afc90c8482f, State: Initialized, Role: FOLLOWER
I20260929 01:06:03.717144 23651 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a7ef3170b3439597cc7afc90c8482f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260929 01:06:03.717383 23651 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:03.717518 23651 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:03.717697 23651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:03.720254 23651 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a7ef3170b3439597cc7afc90c8482f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260929 01:06:03.720805 23651 leader_election.cc:302] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a7ef3170b3439597cc7afc90c8482f; no voters:
I20260929 01:06:03.721735 23651 leader_election.cc:288] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:03.721861 23660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:03.724303 23651 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:03.725412 23660 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [term 1 LEADER]: Becoming Leader. State: Replica: f0a7ef3170b3439597cc7afc90c8482f, State: Running, Role: LEADER
I20260929 01:06:03.726305 23660 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a7ef3170b3439597cc7afc90c8482f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260929 01:06:03.729421 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a7ef3170b3439597cc7afc90c8482f. Latest consensus state: current_term: 1 leader_uuid: "f0a7ef3170b3439597cc7afc90c8482f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a7ef3170b3439597cc7afc90c8482f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } } }
I20260929 01:06:03.729894 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:03.730512 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a7ef3170b3439597cc7afc90c8482f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a7ef3170b3439597cc7afc90c8482f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } } }
I20260929 01:06:03.730827 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:03.736660 23671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:03.742146 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:03.721179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60901 (local address 127.0.0.1:46075)
0929 01:06:03.721398 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:03.721407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:03.722008 (+ 601us) server_negotiation.cc:407] Connection header received
0929 01:06:03.722300 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:03.722310 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:03.722410 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:03.722544 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:03.724843 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:03.726339 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:03.728313 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:03.729123 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:03.729251 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:03.733292 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:03.733302 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:03.733318 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:03.733378 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:03.738203 (+ 4825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:03.739365 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:03.739371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:03.739374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:03.739586 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:03.740168 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:03.740174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:03.740177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:03.740667 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:03.741001 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:03.741481 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 01:06:03.741711 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":68}
I20260929 01:06:03.742705 23671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:03.752416 23671 catalog_manager.cc:1347] Generated new cluster ID: c0bfbd466e4b41a6821303d5636eff7e
I20260929 01:06:03.752558 23671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:03.764452 23671 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:03.766450 23671 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:03.781270 23671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a7ef3170b3439597cc7afc90c8482f: Generated new TSK 0
I20260929 01:06:03.782570 23671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:03.798240 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23584
I20260929 01:06:03.799958 23646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:03.897529 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23584
W20260929 01:06:03.915802 23569 connection.cc:535] client connection to 127.0.0.1:46075 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46075): Connection reset by peer (error 104)
2026-09-29T01:06:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2171 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:04 dist-test-slave-k0b7 krb5kdc[23684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:04 dist-test-slave-k0b7 krb5kdc[23684](info): set up 2 sockets
Sep 29 01:06:04 dist-test-slave-k0b7 krb5kdc[23684](info): commencing operation
krb5kdc: starting...
W20260929 01:06:05.087673 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:05 dist-test-slave-k0b7 krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:05.386874 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:05.753413 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:05.753795 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:05.753975 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:05.761977 23697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:05.762568 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:05.762708 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:05.762837 23697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:05.762944 23697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:05.773634 23697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:05.775411 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:05.776783 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:05.791646 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.791759 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.792460 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:05.792918 23697 server_base.cc:1034] running on GCE node
I20260929 01:06:05.793980 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:05.796195 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:05.799713 23697 hybrid_clock.cc:648] HybridClock initialized: now 1790643965799629 us; error 185 us; skew 500 ppm
I20260929 01:06:05.800419 23697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:05 dist-test-slave-k0b7 krb5kdc[23684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:05.811451 23697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:05.813386 23697 webserver.cc:460] Webserver started at http://127.0.0.1:35325/ using document root <none> and password file <none>
I20260929 01:06:05.814129 23697 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:05.814244 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:05.814617 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:05.817924 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "597754906a3343abbd274d57dff3363e"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-k0b7"
server_key: "0b76510a6be816105b461dbc94d3be51"
server_key_iv: "c4c9c5aa7baa10ea57d497320e84a98d"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.818848 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "597754906a3343abbd274d57dff3363e"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-k0b7"
server_key: "0b76510a6be816105b461dbc94d3be51"
server_key_iv: "c4c9c5aa7baa10ea57d497320e84a98d"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.825121 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:05.829993 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.832099 23697 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:06:05.832369 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "597754906a3343abbd274d57dff3363e"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-k0b7"
server_key: "0b76510a6be816105b461dbc94d3be51"
server_key_iv: "c4c9c5aa7baa10ea57d497320e84a98d"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.832612 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:05.873522 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:05.879962 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:05.880268 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:05.903401 23697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39511
I20260929 01:06:05.903469 23763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39511 every 8 connection(s)
I20260929 01:06:05.905295 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:05.906970 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23697
I20260929 01:06:05.907203 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:05.907795 3159 external_mini_cluster.cc:1428] Setting key 215c7b2041c23c3a716c3796bef9947b
I20260929 01:06:05.911039 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:05.924124 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.910035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52919 (local address 127.0.0.1:39511)
0929 01:06:05.911156 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.911176 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.911234 (+ 58us) server_negotiation.cc:407] Connection header received
0929 01:06:05.913101 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.913127 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.913551 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.913999 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:05.914642 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.916258 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.918280 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.919116 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.919273 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.921001 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:05.921014 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:05.921032 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:05.921853 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:05.921966 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:05.922001 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:05.922181 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.922637 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.922887 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":578,"threads_started":1}
I20260929 01:06:05.924724 23764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e: Bootstrap starting.
I20260929 01:06:05.928308 23764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:05.928737 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:05.929503 23764 log.cc:824] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:05.933156 23764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e: No bootstrap required, opened a new log
I20260929 01:06:05.943146 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.934624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40797 (local address 127.0.0.1:39511)
0929 01:06:05.934848 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.934854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.935291 (+ 437us) server_negotiation.cc:407] Connection header received
0929 01:06:05.935484 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.935489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.935585 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.935726 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:05.936447 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.937662 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.939742 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.940434 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.940564 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.942008 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:05.942014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:05.942021 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:05.942046 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:05.942217 (+ 171us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:05.942238 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:05.942351 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.942653 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.942748 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:06:05.943964 23764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597754906a3343abbd274d57dff3363e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260929 01:06:05.944407 23764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:05.944523 23764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597754906a3343abbd274d57dff3363e, State: Initialized, Role: FOLLOWER
I20260929 01:06:05.945156 23764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597754906a3343abbd274d57dff3363e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260929 01:06:05.945429 23764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:05.945582 23764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:05.945781 23764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:05.948149 23764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597754906a3343abbd274d57dff3363e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260929 01:06:05.948719 23764 leader_election.cc:302] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 597754906a3343abbd274d57dff3363e; no voters:
I20260929 01:06:05.949453 23764 leader_election.cc:288] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:05.949612 23774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:05.950834 23774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [term 1 LEADER]: Becoming Leader. State: Replica: 597754906a3343abbd274d57dff3363e, State: Running, Role: LEADER
I20260929 01:06:05.951511 23774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597754906a3343abbd274d57dff3363e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } }
I20260929 01:06:05.951853 23764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:05.955221 23775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "597754906a3343abbd274d57dff3363e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597754906a3343abbd274d57dff3363e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } } }
I20260929 01:06:05.955199 23776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 597754906a3343abbd274d57dff3363e. Latest consensus state: current_term: 1 leader_uuid: "597754906a3343abbd274d57dff3363e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597754906a3343abbd274d57dff3363e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39511 } } }
I20260929 01:06:05.955677 23775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:05.955749 23776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:05.956626 23783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:05.963016 23783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:05.973419 23783 catalog_manager.cc:1347] Generated new cluster ID: 6df8f7682cc5430aa9166692216013e6
I20260929 01:06:05.973567 23783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:05.988687 23783 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:05.990605 23783 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:05.999750 23783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 597754906a3343abbd274d57dff3363e: Generated new TSK 0
I20260929 01:06:06.001036 23783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:06.021994 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23697
W20260929 01:06:06.024003 23759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52919
W20260929 01:06:06.025153 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:06.025378 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23697
W20260929 01:06:06.044221 23682 connection.cc:535] client connection to 127.0.0.1:39511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39511): Connection reset by peer (error 104)
2026-09-29T01:06:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2129 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:06 dist-test-slave-k0b7 krb5kdc[23795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:06 dist-test-slave-k0b7 krb5kdc[23795](info): set up 2 sockets
Sep 29 01:06:06 dist-test-slave-k0b7 krb5kdc[23795](info): commencing operation
krb5kdc: starting...
W20260929 01:06:07.208232 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:07 dist-test-slave-k0b7 krb5kdc[23795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:07.511214 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:07.865170 23808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:07.865559 23808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:07.865770 23808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:07.873715 23808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:07.873914 23808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:07.874030 23808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:07.874142 23808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:07.874225 23808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:07.884817 23808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:07.886217 23808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:07.887641 23808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:07.900430 23813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:07.900393 23816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:07.900429 23814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:07.901345 23808 server_base.cc:1034] running on GCE node
I20260929 01:06:07.902245 23808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:07.903920 23808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:07.905172 23808 hybrid_clock.cc:648] HybridClock initialized: now 1790643967905100 us; error 97 us; skew 500 ppm
I20260929 01:06:07.905568 23808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:07 dist-test-slave-k0b7 krb5kdc[23795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:07.908905 23808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:07.910669 23808 webserver.cc:460] Webserver started at http://127.0.0.1:43501/ using document root <none> and password file <none>
I20260929 01:06:07.911437 23808 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:07.911592 23808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:07.911914 23808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:07.915267 23808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "95a9e4ed586a48ab9721a4a5b47639df"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-k0b7"
server_key: "c852f9ef2c13152972f53eb8c50d4f25"
server_key_iv: "9f9a4d0429bfea925a42543e2f4ea04a"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.916401 23808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "95a9e4ed586a48ab9721a4a5b47639df"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-k0b7"
server_key: "c852f9ef2c13152972f53eb8c50d4f25"
server_key_iv: "9f9a4d0429bfea925a42543e2f4ea04a"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.922410 23808 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:07.926846 23823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:07.928578 23808 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:06:07.928889 23808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "95a9e4ed586a48ab9721a4a5b47639df"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-k0b7"
server_key: "c852f9ef2c13152972f53eb8c50d4f25"
server_key_iv: "9f9a4d0429bfea925a42543e2f4ea04a"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.929144 23808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:07.935773 23808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:07.941332 23808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:07.941728 23808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:07.963508 23808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33401
I20260929 01:06:07.963567 23874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33401 every 8 connection(s)
I20260929 01:06:07.965025 23808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:07.969163 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:07.969885 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23808
I20260929 01:06:07.970064 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:07.970710 3159 external_mini_cluster.cc:1428] Setting key e278d3c506393f0358df1492ef27650f
I20260929 01:06:07.979964 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:07.972813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36195 (local address 127.0.0.1:33401)
0929 01:06:07.973848 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:07.973860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:07.973913 (+ 53us) server_negotiation.cc:407] Connection header received
0929 01:06:07.975973 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:07.976003 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:07.976450 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:07.976873 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:07.978096 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:07.978103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:07.978117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:07.978135 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:07.978208 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:07.978231 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:07.978432 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:07.978621 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:06:07.978872 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":562,"threads_started":1}
I20260929 01:06:07.984229 23875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df: Bootstrap starting.
I20260929 01:06:07.984308 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:07.988390 23875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:07.989832 23875 log.cc:824] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:07.992790 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:07.989960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59761 (local address 127.0.0.1:33401)
0929 01:06:07.990187 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:07.990195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:07.990809 (+ 614us) server_negotiation.cc:407] Connection header received
0929 01:06:07.991024 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:07.991028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:07.991083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:07.991194 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:07.992144 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:07.992149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:07.992154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:07.992171 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:07.992218 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:07.992233 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:07.992353 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:07.992443 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 01:06:07.992509 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:06:07.993201 23875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df: No bootstrap required, opened a new log
I20260929 01:06:08.001363 23875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a9e4ed586a48ab9721a4a5b47639df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33401 } }
I20260929 01:06:08.001770 23875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:08.001837 23875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a9e4ed586a48ab9721a4a5b47639df, State: Initialized, Role: FOLLOWER
I20260929 01:06:08.002377 23875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a9e4ed586a48ab9721a4a5b47639df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33401 } }
I20260929 01:06:08.002660 23875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:08.002763 23875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:08.002893 23875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:08.005489 23875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a9e4ed586a48ab9721a4a5b47639df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33401 } }
I20260929 01:06:08.006057 23875 leader_election.cc:302] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a9e4ed586a48ab9721a4a5b47639df; no voters:
I20260929 01:06:08.006840 23875 leader_election.cc:288] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:08.007036 23885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:08.008849 23885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [term 1 LEADER]: Becoming Leader. State: Replica: 95a9e4ed586a48ab9721a4a5b47639df, State: Running, Role: LEADER
I20260929 01:06:08.009670 23875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:08.009625 23885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a9e4ed586a48ab9721a4a5b47639df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33401 } }
I20260929 01:06:08.013573 23886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95a9e4ed586a48ab9721a4a5b47639df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a9e4ed586a48ab9721a4a5b47639df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33401 } } }
I20260929 01:06:08.014133 23886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:08.013566 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a9e4ed586a48ab9721a4a5b47639df. Latest consensus state: current_term: 1 leader_uuid: "95a9e4ed586a48ab9721a4a5b47639df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a9e4ed586a48ab9721a4a5b47639df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33401 } } }
I20260929 01:06:08.014973 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:08.017410 23893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:08.023496 23893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:08.033236 23893 catalog_manager.cc:1347] Generated new cluster ID: 60161fe324fa4e7ab35e7238bed9a7ef
I20260929 01:06:08.033372 23893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:08.049391 23893 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:08.049978 23893 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:08.072925 23893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95a9e4ed586a48ab9721a4a5b47639df: Generated new TSK 0
I20260929 01:06:08.074054 23893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:08.090862 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23808
W20260929 01:06:08.092170 23870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36195
W20260929 01:06:08.093078 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23808: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:08.093241 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23808
2026-09-29T01:06:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:08 dist-test-slave-k0b7 krb5kdc[23906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:08 dist-test-slave-k0b7 krb5kdc[23906](info): set up 2 sockets
Sep 29 01:06:08 dist-test-slave-k0b7 krb5kdc[23906](info): commencing operation
krb5kdc: starting...
W20260929 01:06:09.279538 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:09 dist-test-slave-k0b7 krb5kdc[23906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:09.575333 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:09.941427 23919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:09.941751 23919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:09.941922 23919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:09.949796 23919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:09.949941 23919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:09.950024 23919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:09.950145 23919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:09.950238 23919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:09.960852 23919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:09.962354 23919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:09.963794 23919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:09.977146 23927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:09.977272 23925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:09.977263 23924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.978093 23919 server_base.cc:1034] running on GCE node
I20260929 01:06:09.979038 23919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:09.980638 23919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:09.981901 23919 hybrid_clock.cc:648] HybridClock initialized: now 1790643969981823 us; error 68 us; skew 500 ppm
I20260929 01:06:09.982402 23919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:09 dist-test-slave-k0b7 krb5kdc[23906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:09.986028 23919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:09.987690 23919 webserver.cc:460] Webserver started at http://127.0.0.1:46185/ using document root <none> and password file <none>
I20260929 01:06:09.988418 23919 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:09.988521 23919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:09.988788 23919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:09.992000 23919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "39fc222977134455b2f6af1828c80b5d"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-k0b7"
server_key: "9a27eef92b49191c002db862a6336f52"
server_key_iv: "761d6ac2ff8edd258bbf36e72ca54727"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.992769 23919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "39fc222977134455b2f6af1828c80b5d"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-k0b7"
server_key: "9a27eef92b49191c002db862a6336f52"
server_key_iv: "761d6ac2ff8edd258bbf36e72ca54727"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.998474 23919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:06:10.002861 23934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:10.004762 23919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:06:10.005014 23919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "39fc222977134455b2f6af1828c80b5d"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-k0b7"
server_key: "9a27eef92b49191c002db862a6336f52"
server_key_iv: "761d6ac2ff8edd258bbf36e72ca54727"
server_key_version: "encryptionkey@0"
I20260929 01:06:10.005268 23919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:10.041865 23919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:10.047365 23919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:10.047703 23919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:10.074347 23919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38899
I20260929 01:06:10.074393 23985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38899 every 8 connection(s)
I20260929 01:06:10.076845 23919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:10.082759 23986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:10.084681 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 23919
I20260929 01:06:10.084894 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:10.085501 3159 external_mini_cluster.cc:1428] Setting key b00dc4d3016333362a0792488c194578
I20260929 01:06:10.103710 23986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d: Bootstrap starting.
I20260929 01:06:10.108592 23986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:10.109884 23986 log.cc:824] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:06:10 dist-test-slave-k0b7 krb5kdc[23906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:10.114674 23986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d: No bootstrap required, opened a new log
I20260929 01:06:10.123450 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:10.088038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44657 (local address 127.0.0.1:38899)
0929 01:06:10.089276 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:10.089296 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:10.089367 (+ 71us) server_negotiation.cc:407] Connection header received
0929 01:06:10.092206 (+ 2839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:10.092248 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:10.093003 (+ 755us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:10.093582 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:10.097597 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:10.100598 (+ 3001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:10.103447 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:10.105007 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:10.105326 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:10.113449 (+ 8123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:10.113462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:10.113485 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:10.113544 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:10.118335 (+ 4791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:10.119259 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:10.119273 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:10.119283 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:10.119546 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:10.120003 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:10.120008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:10.120012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:10.120709 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:10.121058 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:10.121619 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 01:06:10.122145 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":563,"threads_started":1}
I20260929 01:06:10.128502 23986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fc222977134455b2f6af1828c80b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260929 01:06:10.128942 23986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:10.129084 23986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fc222977134455b2f6af1828c80b5d, State: Initialized, Role: FOLLOWER
I20260929 01:06:10.129900 23986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fc222977134455b2f6af1828c80b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260929 01:06:10.130283 23986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:10.130455 23986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:10.130743 23986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:10.130924 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:10.133460 23986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fc222977134455b2f6af1828c80b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260929 01:06:10.134167 23986 leader_election.cc:302] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39fc222977134455b2f6af1828c80b5d; no voters:
I20260929 01:06:10.134995 23986 leader_election.cc:288] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:10.135356 23994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:10.137919 23986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:10.145077 23994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [term 1 LEADER]: Becoming Leader. State: Replica: 39fc222977134455b2f6af1828c80b5d, State: Running, Role: LEADER
I20260929 01:06:10.145987 23994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fc222977134455b2f6af1828c80b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } }
I20260929 01:06:10.149744 23996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39fc222977134455b2f6af1828c80b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fc222977134455b2f6af1828c80b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } } }
I20260929 01:06:10.149235 23997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39fc222977134455b2f6af1828c80b5d. Latest consensus state: current_term: 1 leader_uuid: "39fc222977134455b2f6af1828c80b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fc222977134455b2f6af1828c80b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38899 } } }
I20260929 01:06:10.150350 23996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:10.150532 23997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:10.151520 24010 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:06:10.152621 24011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:10.157284 24011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:10.158888 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:10.137186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47205 (local address 127.0.0.1:38899)
0929 01:06:10.137888 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:10.137899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:10.138755 (+ 856us) server_negotiation.cc:407] Connection header received
0929 01:06:10.138995 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:10.139001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:10.139087 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:10.139226 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:10.141060 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:10.142406 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:10.144656 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:10.145988 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:10.146182 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:10.150443 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:10.150452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:10.150457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:10.150561 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:10.154608 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:10.155848 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:10.155854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:10.155858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:10.156072 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:10.156691 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:10.156697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:10.156701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:10.157272 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:10.157554 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:10.158110 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 01:06:10.158408 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:06:10.167532 24011 catalog_manager.cc:1347] Generated new cluster ID: da117e9f1d4b4f468fafcb2859bac1ec
I20260929 01:06:10.167683 24011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:10.180442 24011 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:10.181916 24011 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:10.189476 24011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39fc222977134455b2f6af1828c80b5d: Generated new TSK 0
I20260929 01:06:10.190452 24011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:10.205443 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 23919
I20260929 01:06:10.206737 23981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:10.314080 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 23919
2026-09-29T01:06:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:10 dist-test-slave-k0b7 krb5kdc[24018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:10 dist-test-slave-k0b7 krb5kdc[24018](info): set up 2 sockets
Sep 29 01:06:10 dist-test-slave-k0b7 krb5kdc[24018](info): commencing operation
krb5kdc: starting...
W20260929 01:06:11.507109 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.011s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:11 dist-test-slave-k0b7 krb5kdc[24018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:11.811415 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:12.180792 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:12.181115 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:12.181399 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:12.189380 24031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:12.189512 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:12.189587 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:12.189683 24031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:12.189766 24031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:12.200438 24031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:12.201860 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:12.203262 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:12.217052 24036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.217109 24037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.217088 24039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.217813 24031 server_base.cc:1034] running on GCE node
I20260929 01:06:12.218815 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:12.220584 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:12.221884 24031 hybrid_clock.cc:648] HybridClock initialized: now 1790643972221807 us; error 56 us; skew 500 ppm
I20260929 01:06:12.222275 24031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:12 dist-test-slave-k0b7 krb5kdc[24018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:12.235296 24031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:12.237475 24031 webserver.cc:460] Webserver started at http://127.0.0.1:45599/ using document root <none> and password file <none>
I20260929 01:06:12.239006 24031 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:12.239329 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:12.239748 24031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:12.243567 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "403ad70155ad4ff9b011f1a08d5be790"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-k0b7"
server_key: "7ee9ea02ed94ad3e73b89ef79a6de827"
server_key_iv: "94c366fa95160ef1a6fb34fda180ba51"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.244460 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "403ad70155ad4ff9b011f1a08d5be790"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-k0b7"
server_key: "7ee9ea02ed94ad3e73b89ef79a6de827"
server_key_iv: "94c366fa95160ef1a6fb34fda180ba51"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.250730 24031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 01:06:12.255155 24046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.256902 24031 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:06:12.257161 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "403ad70155ad4ff9b011f1a08d5be790"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-k0b7"
server_key: "7ee9ea02ed94ad3e73b89ef79a6de827"
server_key_iv: "94c366fa95160ef1a6fb34fda180ba51"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.257404 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:12.288121 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:12.294018 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:12.294401 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:12.317744 24031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35641
I20260929 01:06:12.317775 24097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35641 every 8 connection(s)
I20260929 01:06:12.319416 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:12.322032 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24031
I20260929 01:06:12.322249 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:12.322801 3159 external_mini_cluster.cc:1428] Setting key 54c3c028c7be87145992b4ddb047c20d
I20260929 01:06:12.325603 24098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:12.337214 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.324881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52043 (local address 127.0.0.1:35641)
0929 01:06:12.325881 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.325894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.325938 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:06:12.327707 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.327728 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.328147 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.328588 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.329191 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.330663 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.332061 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.332839 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.333068 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.334545 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.334558 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.334586 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.335211 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.335299 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.335330 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.335482 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.335731 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.336091 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":387,"threads_started":1}
I20260929 01:06:12.338868 24098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790: Bootstrap starting.
I20260929 01:06:12.341457 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:12.342442 24098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:12.343559 24098 log.cc:824] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:12.347600 24098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790: No bootstrap required, opened a new log
I20260929 01:06:12.356593 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.347078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42947 (local address 127.0.0.1:35641)
0929 01:06:12.347367 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.347375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.347796 (+ 421us) server_negotiation.cc:407] Connection header received
0929 01:06:12.348094 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.348098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.348170 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.348268 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.348994 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.350402 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.353109 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.353583 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.353722 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.355281 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.355289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.355293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.355323 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.355396 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.355421 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.355516 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.355926 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.356082 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 01:06:12.360061 24098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403ad70155ad4ff9b011f1a08d5be790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260929 01:06:12.360551 24098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:12.360728 24098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403ad70155ad4ff9b011f1a08d5be790, State: Initialized, Role: FOLLOWER
I20260929 01:06:12.361413 24098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403ad70155ad4ff9b011f1a08d5be790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260929 01:06:12.361726 24098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:12.361898 24098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:12.362092 24098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:12.364509 24098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403ad70155ad4ff9b011f1a08d5be790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260929 01:06:12.365077 24098 leader_election.cc:302] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403ad70155ad4ff9b011f1a08d5be790; no voters:
I20260929 01:06:12.365850 24098 leader_election.cc:288] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:12.366092 24108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:12.366935 24108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [term 1 LEADER]: Becoming Leader. State: Replica: 403ad70155ad4ff9b011f1a08d5be790, State: Running, Role: LEADER
I20260929 01:06:12.367586 24108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403ad70155ad4ff9b011f1a08d5be790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260929 01:06:12.368536 24098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:12.370985 24110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403ad70155ad4ff9b011f1a08d5be790. Latest consensus state: current_term: 1 leader_uuid: "403ad70155ad4ff9b011f1a08d5be790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403ad70155ad4ff9b011f1a08d5be790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } } }
I20260929 01:06:12.371002 24109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "403ad70155ad4ff9b011f1a08d5be790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403ad70155ad4ff9b011f1a08d5be790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } } }
I20260929 01:06:12.371381 24110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:12.371390 24109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:12.373383 24115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:12.379200 24115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:12.390591 24115 catalog_manager.cc:1347] Generated new cluster ID: 17afafc80f2244b3bd4986e2998b5132
I20260929 01:06:12.390741 24115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:12.419930 24115 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:12.421686 24115 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:12.429733 24115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 403ad70155ad4ff9b011f1a08d5be790: Generated new TSK 0
I20260929 01:06:12.430887 24115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:12.439625 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24031
W20260929 01:06:12.441188 24093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52043
W20260929 01:06:12.442204 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:12.442441 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24031
2026-09-29T01:06:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2127 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:12 dist-test-slave-k0b7 krb5kdc[24129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:12 dist-test-slave-k0b7 krb5kdc[24129](info): set up 2 sockets
Sep 29 01:06:12 dist-test-slave-k0b7 krb5kdc[24129](info): commencing operation
krb5kdc: starting...
W20260929 01:06:13.599582 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:13 dist-test-slave-k0b7 krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:13.895135 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:14.258157 24142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:14.258636 24142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:14.259058 24142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:14.267509 24142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:14.267704 24142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:14.267836 24142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:14.267952 24142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:14.268033 24142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:14.279769 24142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:14.281258 24142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:14.282727 24142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:14.296144 24147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:14.296305 24148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:14.296368 24150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:14.298161 24142 server_base.cc:1034] running on GCE node
I20260929 01:06:14.299273 24142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:14.300949 24142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:14.302197 24142 hybrid_clock.cc:648] HybridClock initialized: now 1790643974302139 us; error 48 us; skew 500 ppm
I20260929 01:06:14.302803 24142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:14 dist-test-slave-k0b7 krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:14.306826 24142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:14.308820 24142 webserver.cc:460] Webserver started at http://127.0.0.1:39749/ using document root <none> and password file <none>
I20260929 01:06:14.309639 24142 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:14.309760 24142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:14.310155 24142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:14.313549 24142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "69bf4df3e2044b03b4fbe870422598b8"
format_stamp: "Formatted at 2026-09-29 01:06:14 on dist-test-slave-k0b7"
server_key: "692a2ed31c24a677a16046dab42928a8"
server_key_iv: "d9115cbd4d5a0dba641ae43d7403688b"
server_key_version: "encryptionkey@0"
I20260929 01:06:14.314530 24142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "69bf4df3e2044b03b4fbe870422598b8"
format_stamp: "Formatted at 2026-09-29 01:06:14 on dist-test-slave-k0b7"
server_key: "692a2ed31c24a677a16046dab42928a8"
server_key_iv: "d9115cbd4d5a0dba641ae43d7403688b"
server_key_version: "encryptionkey@0"
I20260929 01:06:14.320415 24142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:06:14.324447 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:14.326011 24142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:06:14.326331 24142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "69bf4df3e2044b03b4fbe870422598b8"
format_stamp: "Formatted at 2026-09-29 01:06:14 on dist-test-slave-k0b7"
server_key: "692a2ed31c24a677a16046dab42928a8"
server_key_iv: "d9115cbd4d5a0dba641ae43d7403688b"
server_key_version: "encryptionkey@0"
I20260929 01:06:14.326598 24142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:14.333292 24142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:14.337973 24142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:14.338318 24142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:14.360189 24142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32791
I20260929 01:06:14.360500 24208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32791 every 8 connection(s)
I20260929 01:06:14.361737 24142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:14.365105 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24142
I20260929 01:06:14.365247 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:14.365764 3159 external_mini_cluster.cc:1428] Setting key 430004f9360e8c5d8b4a6cf09e030282
I20260929 01:06:14.366361 24209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:14.374538 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:14.367767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42735 (local address 127.0.0.1:32791)
0929 01:06:14.368675 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:14.368688 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:14.368732 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:06:14.370411 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:14.370436 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:14.371023 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:14.371473 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:14.372595 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:14.372602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:14.372615 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:14.372629 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:14.372704 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:14.372733 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:14.373028 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:14.373185 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 01:06:14.373400 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":439,"threads_started":1}
I20260929 01:06:14.378912 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:14.381925 24209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8: Bootstrap starting.
I20260929 01:06:14.386268 24209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:14.386463 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:14.383558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60825 (local address 127.0.0.1:32791)
0929 01:06:14.383821 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:14.383827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:14.384303 (+ 476us) server_negotiation.cc:407] Connection header received
0929 01:06:14.384452 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:14.384456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:14.384509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:14.384610 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:14.385598 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:14.385602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:14.385605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:14.385622 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:14.385665 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:14.385681 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:14.385783 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:14.385975 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 01:06:14.386028 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:06:14.387717 24209 log.cc:824] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:14.391016 24209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8: No bootstrap required, opened a new log
I20260929 01:06:14.399260 24209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf4df3e2044b03b4fbe870422598b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260929 01:06:14.399744 24209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:14.399863 24209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bf4df3e2044b03b4fbe870422598b8, State: Initialized, Role: FOLLOWER
I20260929 01:06:14.400628 24209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf4df3e2044b03b4fbe870422598b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260929 01:06:14.400942 24209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:14.401129 24209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:14.401340 24209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:14.403865 24209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf4df3e2044b03b4fbe870422598b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260929 01:06:14.404448 24209 leader_election.cc:302] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69bf4df3e2044b03b4fbe870422598b8; no voters:
I20260929 01:06:14.405061 24209 leader_election.cc:288] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:14.405329 24219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:14.406173 24219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [term 1 LEADER]: Becoming Leader. State: Replica: 69bf4df3e2044b03b4fbe870422598b8, State: Running, Role: LEADER
I20260929 01:06:14.406854 24219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf4df3e2044b03b4fbe870422598b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } }
I20260929 01:06:14.407555 24209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:14.410454 24221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69bf4df3e2044b03b4fbe870422598b8. Latest consensus state: current_term: 1 leader_uuid: "69bf4df3e2044b03b4fbe870422598b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf4df3e2044b03b4fbe870422598b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } } }
I20260929 01:06:14.410458 24220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69bf4df3e2044b03b4fbe870422598b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf4df3e2044b03b4fbe870422598b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32791 } } }
I20260929 01:06:14.411175 24221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:14.411218 24220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:14.412194 24228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:14.420002 24228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:14.430375 24228 catalog_manager.cc:1347] Generated new cluster ID: 02b9b606ddeb4f6599c38b50c0f15a7e
I20260929 01:06:14.430550 24228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:14.448705 24228 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:14.449214 24228 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:14.458657 24228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69bf4df3e2044b03b4fbe870422598b8: Generated new TSK 0
I20260929 01:06:14.459630 24228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:14.474366 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24142
W20260929 01:06:14.475732 24204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42735
W20260929 01:06:14.476621 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:14.476786 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24142
2026-09-29T01:06:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:14 dist-test-slave-k0b7 krb5kdc[24240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:14 dist-test-slave-k0b7 krb5kdc[24240](info): set up 2 sockets
Sep 29 01:06:14 dist-test-slave-k0b7 krb5kdc[24240](info): commencing operation
krb5kdc: starting...
W20260929 01:06:15.651908 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:15 dist-test-slave-k0b7 krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:15.942685 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:16.300428 24253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:16.300748 24253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:16.300927 24253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:16.309355 24253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:16.309517 24253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:16.309651 24253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:16.309783 24253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:16.309890 24253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:16.321413 24253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:16.322860 24253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:16.324216 24253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:16.338186 24258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:16.338445 24259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:16.338411 24261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:16.338759 24253 server_base.cc:1034] running on GCE node
I20260929 01:06:16.340317 24253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:16.342182 24253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:16.343493 24253 hybrid_clock.cc:648] HybridClock initialized: now 1790643976343411 us; error 81 us; skew 500 ppm
I20260929 01:06:16.343969 24253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:16 dist-test-slave-k0b7 krb5kdc[24240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:16.348237 24253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:16.350160 24253 webserver.cc:460] Webserver started at http://127.0.0.1:41661/ using document root <none> and password file <none>
I20260929 01:06:16.351141 24253 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:16.351312 24253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:16.351648 24253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:16.354938 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "53a76029d2d94feb80e3bb83ea5b6ea9"
format_stamp: "Formatted at 2026-09-29 01:06:16 on dist-test-slave-k0b7"
server_key: "0259438dbbdf1e408ba9419dafc411da"
server_key_iv: "59684a995017e13984d787df8df74b94"
server_key_version: "encryptionkey@0"
I20260929 01:06:16.355821 24253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "53a76029d2d94feb80e3bb83ea5b6ea9"
format_stamp: "Formatted at 2026-09-29 01:06:16 on dist-test-slave-k0b7"
server_key: "0259438dbbdf1e408ba9419dafc411da"
server_key_iv: "59684a995017e13984d787df8df74b94"
server_key_version: "encryptionkey@0"
I20260929 01:06:16.361603 24253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:06:16.365842 24268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:16.367823 24253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:06:16.368040 24253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "53a76029d2d94feb80e3bb83ea5b6ea9"
format_stamp: "Formatted at 2026-09-29 01:06:16 on dist-test-slave-k0b7"
server_key: "0259438dbbdf1e408ba9419dafc411da"
server_key_iv: "59684a995017e13984d787df8df74b94"
server_key_version: "encryptionkey@0"
I20260929 01:06:16.368276 24253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:16.383423 24253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:16.388504 24253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:16.388839 24253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:16.410585 24253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42031
I20260929 01:06:16.410601 24319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42031 every 8 connection(s)
I20260929 01:06:16.412117 24253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:16.416411 24320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:16.422219 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24253
I20260929 01:06:16.422416 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:16.422956 3159 external_mini_cluster.cc:1428] Setting key 287369a791f5346aa1836bb785ee3bf0
I20260929 01:06:16.430012 24320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9: Bootstrap starting.
I20260929 01:06:16.433334 24320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:16.434424 24320 log.cc:824] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:16.437618 24320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9: No bootstrap required, opened a new log
Sep 29 01:06:16 dist-test-slave-k0b7 krb5kdc[24240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:16.446616 24320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42031 } }
I20260929 01:06:16.447072 24320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:16.447203 24320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a76029d2d94feb80e3bb83ea5b6ea9, State: Initialized, Role: FOLLOWER
I20260929 01:06:16.447880 24320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42031 } }
I20260929 01:06:16.448139 24320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:16.448336 24320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:16.448531 24320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:16.449903 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:16.425193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:47407 (local address 127.0.0.1:42031)
0929 01:06:16.426603 (+ 1410us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:16.426618 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:16.426682 (+ 64us) server_negotiation.cc:407] Connection header received
0929 01:06:16.429033 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:16.429064 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:16.429658 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:16.430159 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:16.431872 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:16.433313 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:16.434858 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:16.435681 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:16.435945 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:16.440888 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:16.440898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:16.440921 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:16.440970 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:16.444961 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:16.445969 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:16.445976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:16.445984 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:16.446168 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:16.446724 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:16.446728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:16.446730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:16.447452 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:16.447822 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:16.448305 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 01:06:16.448715 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":491,"threads_started":1}
I20260929 01:06:16.451592 24320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42031 } }
I20260929 01:06:16.452145 24320 leader_election.cc:302] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53a76029d2d94feb80e3bb83ea5b6ea9; no voters:
I20260929 01:06:16.452795 24320 leader_election.cc:288] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:16.452981 24325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:16.454782 24325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 53a76029d2d94feb80e3bb83ea5b6ea9, State: Running, Role: LEADER
I20260929 01:06:16.454985 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:16.455138 24320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:16.455447 24325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42031 } }
I20260929 01:06:16.464197 24327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a76029d2d94feb80e3bb83ea5b6ea9. Latest consensus state: current_term: 1 leader_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42031 } } }
I20260929 01:06:16.464785 24327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:16.465590 24326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a76029d2d94feb80e3bb83ea5b6ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42031 } } }
I20260929 01:06:16.465973 24326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:16.470137 24341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:16.476087 24341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:16.489331 24341 catalog_manager.cc:1347] Generated new cluster ID: a918dede0ec44af3a48070a46cef732a
I20260929 01:06:16.489459 24341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:16.496372 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:16.473916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:55525 (local address 127.0.0.1:42031)
0929 01:06:16.474108 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:16.474114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:16.476887 (+ 2773us) server_negotiation.cc:407] Connection header received
0929 01:06:16.477225 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:16.477235 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:16.477334 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:16.477418 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:16.479833 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:16.481069 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:16.483133 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:16.484095 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:16.484244 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:16.488827 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:16.488833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:16.488837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:16.488885 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:16.492856 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:16.493981 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:16.493986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:16.493990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:16.494136 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:16.494723 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:16.494727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:16.494730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:16.495108 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:16.495367 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:16.495810 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 01:06:16.495995 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:06:16.505930 24341 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:16.507352 24341 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:16.514763 24341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53a76029d2d94feb80e3bb83ea5b6ea9: Generated new TSK 0
I20260929 01:06:16.515890 24341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:16.526295 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24253
I20260929 01:06:16.527695 24315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:16.621992 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24253
W20260929 01:06:16.640540 24238 connection.cc:535] client connection to 127.0.0.1:42031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42031): Connection reset by peer (error 104)
2026-09-29T01:06:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:16 dist-test-slave-k0b7 krb5kdc[24353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:16 dist-test-slave-k0b7 krb5kdc[24353](info): set up 2 sockets
Sep 29 01:06:16 dist-test-slave-k0b7 krb5kdc[24353](info): commencing operation
krb5kdc: starting...
W20260929 01:06:17.799243 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:18 dist-test-slave-k0b7 krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:18.128623 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:18.492624 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:18.493018 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:18.493208 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:18.501256 24366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:18.501456 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:18.501578 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:18.501690 24366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:18.501771 24366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:18.512496 24366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:18.514065 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:18.515597 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:18.529083 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:18.529052 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:18.529107 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:18.529755 24366 server_base.cc:1034] running on GCE node
I20260929 01:06:18.530828 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:18.532542 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:18.533887 24366 hybrid_clock.cc:648] HybridClock initialized: now 1790643978533846 us; error 48 us; skew 500 ppm
I20260929 01:06:18.534329 24366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:18 dist-test-slave-k0b7 krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:18.546093 24366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:18.547927 24366 webserver.cc:460] Webserver started at http://127.0.0.1:33211/ using document root <none> and password file <none>
I20260929 01:06:18.548662 24366 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:18.548779 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:18.549078 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:18.552342 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "d1616395a868422d99fc53fe4db33116"
format_stamp: "Formatted at 2026-09-29 01:06:18 on dist-test-slave-k0b7"
server_key: "165dd1cd88c39797230d3af35e0b4bda"
server_key_iv: "aff652fc0a23b494cf40210b91a2f0c2"
server_key_version: "encryptionkey@0"
I20260929 01:06:18.553381 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "d1616395a868422d99fc53fe4db33116"
format_stamp: "Formatted at 2026-09-29 01:06:18 on dist-test-slave-k0b7"
server_key: "165dd1cd88c39797230d3af35e0b4bda"
server_key_iv: "aff652fc0a23b494cf40210b91a2f0c2"
server_key_version: "encryptionkey@0"
I20260929 01:06:18.560608 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:06:18.565064 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:18.567024 24366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:06:18.567359 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "d1616395a868422d99fc53fe4db33116"
format_stamp: "Formatted at 2026-09-29 01:06:18 on dist-test-slave-k0b7"
server_key: "165dd1cd88c39797230d3af35e0b4bda"
server_key_iv: "aff652fc0a23b494cf40210b91a2f0c2"
server_key_version: "encryptionkey@0"
I20260929 01:06:18.567620 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:18.583688 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:18.588620 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:18.588985 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:18.611476 24366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41945
I20260929 01:06:18.611523 24432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41945 every 8 connection(s)
I20260929 01:06:18.613062 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:18.617190 24433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:18.618996 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24366
I20260929 01:06:18.619192 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:18.619714 3159 external_mini_cluster.cc:1428] Setting key 3c77fbe7a2e9bdbd092710d9742161f0
I20260929 01:06:18.631857 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116: Bootstrap starting.
I20260929 01:06:18.634285 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:18.621929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:34035 (local address 127.0.0.1:41945)
0929 01:06:18.623001 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:18.623017 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:18.623067 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:06:18.624820 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:18.624850 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:18.625399 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:18.625843 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:18.626444 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.627926 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.629341 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.630066 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.630275 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:18.631716 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:18.631724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:18.631740 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:18.632439 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:18.632534 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:18.632561 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:18.632740 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:18.633032 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 01:06:18.633211 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":599,"threads_started":1}
I20260929 01:06:18.635478 24433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:18.636623 24433 log.cc:824] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:18.638620 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:18.640527 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116: No bootstrap required, opened a new log
I20260929 01:06:18.651068 24433 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1616395a868422d99fc53fe4db33116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260929 01:06:18.651710 24433 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:18.651947 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:18.644372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:37277 (local address 127.0.0.1:41945)
0929 01:06:18.644589 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:18.644597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:18.645006 (+ 409us) server_negotiation.cc:407] Connection header received
0929 01:06:18.645167 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:18.645174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:18.645268 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:18.645408 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:18.646019 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.647268 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.648756 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.649496 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.649639 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:18.651002 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:18.651009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:18.651013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:18.651039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:18.651118 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:18.651145 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:18.651268 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:18.651503 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 01:06:18.651565 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:06:18.651924 24433 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1616395a868422d99fc53fe4db33116, State: Initialized, Role: FOLLOWER
I20260929 01:06:18.652988 24433 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1616395a868422d99fc53fe4db33116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260929 01:06:18.653286 24433 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:18.653455 24433 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:18.653636 24433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:18.656137 24433 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1616395a868422d99fc53fe4db33116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260929 01:06:18.656721 24433 leader_election.cc:302] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1616395a868422d99fc53fe4db33116; no voters:
I20260929 01:06:18.657559 24433 leader_election.cc:288] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:18.657866 24443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:18.659162 24443 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [term 1 LEADER]: Becoming Leader. State: Replica: d1616395a868422d99fc53fe4db33116, State: Running, Role: LEADER
I20260929 01:06:18.659881 24443 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1616395a868422d99fc53fe4db33116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260929 01:06:18.660074 24433 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:18.663256 24444 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1616395a868422d99fc53fe4db33116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1616395a868422d99fc53fe4db33116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } } }
I20260929 01:06:18.663761 24444 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:18.663280 24445 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1616395a868422d99fc53fe4db33116. Latest consensus state: current_term: 1 leader_uuid: "d1616395a868422d99fc53fe4db33116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1616395a868422d99fc53fe4db33116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } } }
I20260929 01:06:18.664170 24445 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:18.669998 24452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:18.676105 24452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:18.687116 24452 catalog_manager.cc:1347] Generated new cluster ID: b57e7f99d9b341d0b563e01f2fb9f61c
I20260929 01:06:18.687314 24452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:18.708992 24452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:18.711053 24452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:18.721376 24452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1616395a868422d99fc53fe4db33116: Generated new TSK 0
I20260929 01:06:18.722419 24452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:18.740489 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24366
W20260929 01:06:18.741740 24428 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:34035
W20260929 01:06:18.742666 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24366: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:18.742854 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24366
W20260929 01:06:18.762099 24351 connection.cc:535] client connection to 127.0.0.1:41945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41945): Connection reset by peer (error 104)
2026-09-29T01:06:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2121 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:18 dist-test-slave-k0b7 krb5kdc[24464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:18 dist-test-slave-k0b7 krb5kdc[24464](info): set up 2 sockets
Sep 29 01:06:18 dist-test-slave-k0b7 krb5kdc[24464](info): commencing operation
krb5kdc: starting...
W20260929 01:06:19.915236 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:20 dist-test-slave-k0b7 krb5kdc[24464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:20.223714 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:20.591466 24477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:20.591806 24477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:20.591904 24477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:20.599781 24477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:20.599934 24477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:20.599985 24477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:20.600049 24477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:20.600095 24477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:20.610800 24477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:20.612138 24477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:20.613406 24477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:20.626991 24485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:20.627117 24483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:20.627118 24482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:20.628448 24477 server_base.cc:1034] running on GCE node
I20260929 01:06:20.629453 24477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:20.631239 24477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:20.632539 24477 hybrid_clock.cc:648] HybridClock initialized: now 1790643980632476 us; error 54 us; skew 500 ppm
I20260929 01:06:20.633040 24477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:20 dist-test-slave-k0b7 krb5kdc[24464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:20.636926 24477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:20.638934 24477 webserver.cc:460] Webserver started at http://127.0.0.1:40471/ using document root <none> and password file <none>
I20260929 01:06:20.639757 24477 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:20.639961 24477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:20.640347 24477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:20.643873 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "4940fe22ff954c5282812ca03260636f"
format_stamp: "Formatted at 2026-09-29 01:06:20 on dist-test-slave-k0b7"
server_key: "cad53de65089a8d7c67427892236b547"
server_key_iv: "2116f1c71f670db684ce901d7a7d589e"
server_key_version: "encryptionkey@0"
I20260929 01:06:20.644930 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "4940fe22ff954c5282812ca03260636f"
format_stamp: "Formatted at 2026-09-29 01:06:20 on dist-test-slave-k0b7"
server_key: "cad53de65089a8d7c67427892236b547"
server_key_iv: "2116f1c71f670db684ce901d7a7d589e"
server_key_version: "encryptionkey@0"
I20260929 01:06:20.651276 24477 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:06:20.655439 24492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:20.657125 24477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:06:20.657377 24477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "4940fe22ff954c5282812ca03260636f"
format_stamp: "Formatted at 2026-09-29 01:06:20 on dist-test-slave-k0b7"
server_key: "cad53de65089a8d7c67427892236b547"
server_key_iv: "2116f1c71f670db684ce901d7a7d589e"
server_key_version: "encryptionkey@0"
I20260929 01:06:20.657622 24477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:20.664187 24477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:20.669175 24477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:20.669631 24477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:20.692159 24477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43271
I20260929 01:06:20.692211 24543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43271 every 8 connection(s)
I20260929 01:06:20.693763 24477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:20.694615 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24477
I20260929 01:06:20.694802 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:20.695389 3159 external_mini_cluster.cc:1428] Setting key e0ff17cc7aa382fdec5e0da3081c9f6d
I20260929 01:06:20.699218 24544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:20.704011 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:20.697468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:57707 (local address 127.0.0.1:43271)
0929 01:06:20.698408 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:20.698420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:20.698461 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:06:20.700290 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:20.700314 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:20.700649 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:20.701122 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:20.702231 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:20.702237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:20.702251 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:20.702274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:20.702346 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:20.702368 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:20.702673 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:20.702837 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 01:06:20.703023 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":456,"threads_started":1}
I20260929 01:06:20.709067 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:20.712944 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f: Bootstrap starting.
I20260929 01:06:20.716642 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:20.713875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:49421 (local address 127.0.0.1:43271)
0929 01:06:20.714052 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:20.714058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:20.714458 (+ 400us) server_negotiation.cc:407] Connection header received
0929 01:06:20.714684 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:20.714692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:20.714772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:20.714898 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:20.715938 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:20.715944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:20.715950 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:20.715976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:20.716030 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:20.716055 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:20.716187 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:20.716288 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 01:06:20.716364 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:06:20.717090 24544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:20.718397 24544 log.cc:824] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:20.721746 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f: No bootstrap required, opened a new log
I20260929 01:06:20.729964 24544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4940fe22ff954c5282812ca03260636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260929 01:06:20.730367 24544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:20.730513 24544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4940fe22ff954c5282812ca03260636f, State: Initialized, Role: FOLLOWER
I20260929 01:06:20.731158 24544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4940fe22ff954c5282812ca03260636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260929 01:06:20.731375 24544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:20.731506 24544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:20.731693 24544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:20.734090 24544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4940fe22ff954c5282812ca03260636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260929 01:06:20.734730 24544 leader_election.cc:302] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4940fe22ff954c5282812ca03260636f; no voters:
I20260929 01:06:20.735282 24544 leader_election.cc:288] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:20.735555 24554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:20.737005 24554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [term 1 LEADER]: Becoming Leader. State: Replica: 4940fe22ff954c5282812ca03260636f, State: Running, Role: LEADER
I20260929 01:06:20.737754 24544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:20.737668 24554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4940fe22ff954c5282812ca03260636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } }
I20260929 01:06:20.740757 24556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4940fe22ff954c5282812ca03260636f. Latest consensus state: current_term: 1 leader_uuid: "4940fe22ff954c5282812ca03260636f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4940fe22ff954c5282812ca03260636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } } }
I20260929 01:06:20.741142 24556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:20.741408 24555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4940fe22ff954c5282812ca03260636f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4940fe22ff954c5282812ca03260636f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43271 } } }
I20260929 01:06:20.741763 24555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:20.743801 24562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:20.750420 24562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:20.761480 24562 catalog_manager.cc:1347] Generated new cluster ID: a667c2ee2f3c4d1b93a7a93ed218f943
I20260929 01:06:20.761655 24562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:20.779505 24562 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:20.780074 24562 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:20.788335 24562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4940fe22ff954c5282812ca03260636f: Generated new TSK 0
I20260929 01:06:20.789345 24562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:20.806142 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24477
W20260929 01:06:20.807688 24539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:57707
W20260929 01:06:20.808698 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:20.808903 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24477
2026-09-29T01:06:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:20 dist-test-slave-k0b7 krb5kdc[24575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:20 dist-test-slave-k0b7 krb5kdc[24575](info): set up 2 sockets
Sep 29 01:06:20 dist-test-slave-k0b7 krb5kdc[24575](info): commencing operation
krb5kdc: starting...
W20260929 01:06:21.991181 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:22 dist-test-slave-k0b7 krb5kdc[24575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:22.286222 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:22.651711 24588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:22.652134 24588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:22.652320 24588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:22.660321 24588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:22.660514 24588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:22.660631 24588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:22.660750 24588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:22.660880 24588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:22.672775 24588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:22.674285 24588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:22.675729 24588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:22.688946 24593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:22.688922 24596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:22.688877 24594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:22.693039 24588 server_base.cc:1034] running on GCE node
I20260929 01:06:22.693981 24588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:22.695816 24588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:22.697100 24588 hybrid_clock.cc:648] HybridClock initialized: now 1790643982697057 us; error 44 us; skew 500 ppm
I20260929 01:06:22.697610 24588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:22 dist-test-slave-k0b7 krb5kdc[24575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:22.701858 24588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:22.704113 24588 webserver.cc:460] Webserver started at http://127.0.0.1:36285/ using document root <none> and password file <none>
I20260929 01:06:22.704916 24588 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:22.705072 24588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:22.705420 24588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:22.708702 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "b6e4e92cc3d14348ba320edbf3ea7d65"
format_stamp: "Formatted at 2026-09-29 01:06:22 on dist-test-slave-k0b7"
server_key: "bcd33edda190f41d3c1a4102911087d6"
server_key_iv: "0fa78d85a7b0f724fe2e01f4185b1138"
server_key_version: "encryptionkey@0"
I20260929 01:06:22.709580 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "b6e4e92cc3d14348ba320edbf3ea7d65"
format_stamp: "Formatted at 2026-09-29 01:06:22 on dist-test-slave-k0b7"
server_key: "bcd33edda190f41d3c1a4102911087d6"
server_key_iv: "0fa78d85a7b0f724fe2e01f4185b1138"
server_key_version: "encryptionkey@0"
I20260929 01:06:22.716014 24588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 01:06:22.720193 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:22.721902 24588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:22.722131 24588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "b6e4e92cc3d14348ba320edbf3ea7d65"
format_stamp: "Formatted at 2026-09-29 01:06:22 on dist-test-slave-k0b7"
server_key: "bcd33edda190f41d3c1a4102911087d6"
server_key_iv: "0fa78d85a7b0f724fe2e01f4185b1138"
server_key_version: "encryptionkey@0"
I20260929 01:06:22.722371 24588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:22.744182 24588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:22.749699 24588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:22.750119 24588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:22.772521 24588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45027
I20260929 01:06:22.772540 24654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45027 every 8 connection(s)
I20260929 01:06:22.774010 24588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:22.776649 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24588
I20260929 01:06:22.776850 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:22.777431 3159 external_mini_cluster.cc:1428] Setting key 96f914f78bbade3716306b28bb3aadfc
I20260929 01:06:22.779093 24655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:22.794795 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65: Bootstrap starting.
Sep 29 01:06:22 dist-test-slave-k0b7 krb5kdc[24575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:22.799397 24655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:22.801002 24655 log.cc:824] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:22.805392 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65: No bootstrap required, opened a new log
I20260929 01:06:22.808032 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:22.779666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:45117 (local address 127.0.0.1:45027)
0929 01:06:22.780741 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:22.780759 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:22.781224 (+ 465us) server_negotiation.cc:407] Connection header received
0929 01:06:22.784017 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:22.784050 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:22.784692 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:22.785226 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:22.786917 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.788888 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.790622 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.791877 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.792114 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:22.797247 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:22.797261 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:22.797294 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:22.797359 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:22.801936 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.803230 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.803239 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.803248 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.803775 (+ 527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.804391 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.804400 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.804405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.805175 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:22.805541 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:22.806139 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 01:06:22.806574 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":484,"threads_started":1}
I20260929 01:06:22.813282 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:22.816751 24655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260929 01:06:22.817476 24655 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:22.817662 24655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e4e92cc3d14348ba320edbf3ea7d65, State: Initialized, Role: FOLLOWER
I20260929 01:06:22.818753 24655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260929 01:06:22.819195 24655 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:22.819367 24655 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:22.819569 24655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:22.823271 24655 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260929 01:06:22.823927 24655 leader_election.cc:302] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e4e92cc3d14348ba320edbf3ea7d65; no voters:
I20260929 01:06:22.824680 24655 leader_election.cc:288] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:22.824834 24665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:22.825556 24665 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [term 1 LEADER]: Becoming Leader. State: Replica: b6e4e92cc3d14348ba320edbf3ea7d65, State: Running, Role: LEADER
I20260929 01:06:22.826303 24665 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260929 01:06:22.828713 24655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:22.830034 24667 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } } }
I20260929 01:06:22.830468 24667 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:22.831136 24666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e4e92cc3d14348ba320edbf3ea7d65. Latest consensus state: current_term: 1 leader_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e4e92cc3d14348ba320edbf3ea7d65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } } }
I20260929 01:06:22.831538 24666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:22.839126 24670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:22.841949 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:22.819803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:39763 (local address 127.0.0.1:45027)
0929 01:06:22.820055 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:22.820064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:22.820808 (+ 744us) server_negotiation.cc:407] Connection header received
0929 01:06:22.820988 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:22.820993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:22.821070 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:22.821169 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:22.823317 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.824470 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.826835 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.827524 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.827635 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:22.831885 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:22.831897 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:22.831905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:22.831966 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:22.837066 (+ 5100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.838704 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.838713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.838720 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.838940 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.839775 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.839781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.839784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.840391 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:22.840752 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:22.841441 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 01:06:22.841651 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:06:22.846535 24670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:22.859176 24670 catalog_manager.cc:1347] Generated new cluster ID: a63498ff5cfb4e78915ef021831d6339
I20260929 01:06:22.859341 24670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:22.870144 24670 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:22.872457 24670 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:22.890043 24670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6e4e92cc3d14348ba320edbf3ea7d65: Generated new TSK 0
I20260929 01:06:22.891209 24670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:22.904304 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24588
I20260929 01:06:22.905552 24650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:23.003616 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24588
W20260929 01:06:23.021924 24573 connection.cc:535] client connection to 127.0.0.1:45027 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45027): Connection reset by peer (error 104)
2026-09-29T01:06:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:23 dist-test-slave-k0b7 krb5kdc[24688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:23 dist-test-slave-k0b7 krb5kdc[24688](info): set up 2 sockets
Sep 29 01:06:23 dist-test-slave-k0b7 krb5kdc[24688](info): commencing operation
krb5kdc: starting...
W20260929 01:06:24.187850 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:24 dist-test-slave-k0b7 krb5kdc[24688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:24.484643 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:24.849787 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:24.850135 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:24.850317 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:24.858273 24701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:24.858409 24701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:24.858466 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:24.858630 24701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:24.858706 24701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:24.869575 24701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:24.871155 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:24.872610 24701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:24.887140 24706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:24.887125 24707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:24.887235 24709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:24.888204 24701 server_base.cc:1034] running on GCE node
I20260929 01:06:24.889145 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:24.890848 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:24.892145 24701 hybrid_clock.cc:648] HybridClock initialized: now 1790643984892073 us; error 66 us; skew 500 ppm
I20260929 01:06:24.892652 24701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:24 dist-test-slave-k0b7 krb5kdc[24688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:24.896799 24701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:24.898808 24701 webserver.cc:460] Webserver started at http://127.0.0.1:39319/ using document root <none> and password file <none>
I20260929 01:06:24.899590 24701 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:24.899751 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:24.900089 24701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:24.903510 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "f4b12447f3cd441b86b8fed68c44f39d"
format_stamp: "Formatted at 2026-09-29 01:06:24 on dist-test-slave-k0b7"
server_key: "50ff59f3d1ea14aa4278a19e7fa9a29c"
server_key_iv: "99701c6b1ec19d1011c59163aed64fc6"
server_key_version: "encryptionkey@0"
I20260929 01:06:24.904372 24701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "f4b12447f3cd441b86b8fed68c44f39d"
format_stamp: "Formatted at 2026-09-29 01:06:24 on dist-test-slave-k0b7"
server_key: "50ff59f3d1ea14aa4278a19e7fa9a29c"
server_key_iv: "99701c6b1ec19d1011c59163aed64fc6"
server_key_version: "encryptionkey@0"
I20260929 01:06:24.910698 24701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 01:06:24.915051 24716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:24.916893 24701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:06:24.917150 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "f4b12447f3cd441b86b8fed68c44f39d"
format_stamp: "Formatted at 2026-09-29 01:06:24 on dist-test-slave-k0b7"
server_key: "50ff59f3d1ea14aa4278a19e7fa9a29c"
server_key_iv: "99701c6b1ec19d1011c59163aed64fc6"
server_key_version: "encryptionkey@0"
I20260929 01:06:24.917389 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:24.939154 24701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:24.944173 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:24.944495 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:24.965461 24701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40255
I20260929 01:06:24.965505 24767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40255 every 8 connection(s)
I20260929 01:06:24.967039 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:24.970561 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24701
I20260929 01:06:24.970870 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:24.971335 3159 external_mini_cluster.cc:1428] Setting key 7ad573d9fbc03e8068528bb4558388b6
I20260929 01:06:24.971385 24768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:24.987540 24768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d: Bootstrap starting.
I20260929 01:06:24.988260 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:24.973260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:34687 (local address 127.0.0.1:40255)
0929 01:06:24.974091 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:24.974112 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:24.974169 (+ 57us) server_negotiation.cc:407] Connection header received
0929 01:06:24.976732 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:24.976764 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:24.977390 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:24.977882 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:24.978452 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:24.980719 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:24.982243 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:24.983383 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:24.983605 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:24.985143 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:24.985153 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:24.985177 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:24.985936 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:24.986065 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:24.986096 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:24.986255 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:24.986624 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 01:06:24.986864 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":342,"threads_started":1}
I20260929 01:06:24.991127 24768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:24.992318 24768 log.cc:824] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:24.992973 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:24.995256 24768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d: No bootstrap required, opened a new log
I20260929 01:06:25.005424 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:24.997621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:49033 (local address 127.0.0.1:40255)
0929 01:06:24.997837 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:24.997864 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:24.998397 (+ 533us) server_negotiation.cc:407] Connection header received
0929 01:06:24.998628 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:24.998632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:24.998702 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:24.998790 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:24.999511 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:25.000332 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:25.002293 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:25.002833 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:25.003002 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:25.004407 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:25.004413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:25.004422 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:25.004449 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:25.004495 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:25.004516 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:25.004613 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:25.004907 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 01:06:25.005013 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:06:25.006122 24768 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b12447f3cd441b86b8fed68c44f39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260929 01:06:25.006742 24768 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:25.006865 24768 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b12447f3cd441b86b8fed68c44f39d, State: Initialized, Role: FOLLOWER
I20260929 01:06:25.007601 24768 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b12447f3cd441b86b8fed68c44f39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260929 01:06:25.007869 24768 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:25.008028 24768 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:25.008209 24768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:25.010638 24768 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b12447f3cd441b86b8fed68c44f39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260929 01:06:25.011220 24768 leader_election.cc:302] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4b12447f3cd441b86b8fed68c44f39d; no voters:
I20260929 01:06:25.011900 24768 leader_election.cc:288] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:25.012059 24778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:25.012828 24778 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [term 1 LEADER]: Becoming Leader. State: Replica: f4b12447f3cd441b86b8fed68c44f39d, State: Running, Role: LEADER
I20260929 01:06:25.013578 24778 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b12447f3cd441b86b8fed68c44f39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260929 01:06:25.015472 24768 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:25.016733 24780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4b12447f3cd441b86b8fed68c44f39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b12447f3cd441b86b8fed68c44f39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } } }
I20260929 01:06:25.017200 24780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:25.018424 24779 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b12447f3cd441b86b8fed68c44f39d. Latest consensus state: current_term: 1 leader_uuid: "f4b12447f3cd441b86b8fed68c44f39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b12447f3cd441b86b8fed68c44f39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } } }
I20260929 01:06:25.018898 24779 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:25.023020 24783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:25.028627 24783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:25.039527 24783 catalog_manager.cc:1347] Generated new cluster ID: 5ca42cf44f85413f95ce5b90e88f7b6b
I20260929 01:06:25.039757 24783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:25.049770 24783 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:25.051318 24783 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:25.058643 24783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4b12447f3cd441b86b8fed68c44f39d: Generated new TSK 0
I20260929 01:06:25.059545 24783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:25.073671 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24701
W20260929 01:06:25.075057 24763 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:34687
W20260929 01:06:25.075996 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24701: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:25.076179 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24701
W20260929 01:06:25.094189 24686 connection.cc:535] client connection to 127.0.0.1:40255 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40255): Connection reset by peer (error 104)
2026-09-29T01:06:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:25 dist-test-slave-k0b7 krb5kdc[24799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:25 dist-test-slave-k0b7 krb5kdc[24799](info): set up 2 sockets
Sep 29 01:06:25 dist-test-slave-k0b7 krb5kdc[24799](info): commencing operation
krb5kdc: starting...
W20260929 01:06:26.267417 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:26 dist-test-slave-k0b7 krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:26.564371 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:26.936998 24812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:26.937327 24812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:26.937495 24812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:26.945154 24812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:26.945297 24812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:26.945384 24812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:26.945485 24812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:26.945555 24812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:26.956511 24812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:26.957953 24812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:26.959364 24812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:26.973100 24817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:26.973184 24820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:26.973081 24818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:26.973464 24812 server_base.cc:1034] running on GCE node
I20260929 01:06:26.974797 24812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:26.976513 24812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:26.978094 24812 hybrid_clock.cc:648] HybridClock initialized: now 1790643986978042 us; error 42 us; skew 500 ppm
I20260929 01:06:26.978663 24812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:26 dist-test-slave-k0b7 krb5kdc[24799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:26.982746 24812 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:26.984695 24812 webserver.cc:460] Webserver started at http://127.0.0.1:32859/ using document root <none> and password file <none>
I20260929 01:06:26.985503 24812 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:26.985678 24812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:26.986028 24812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:26.989488 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856"
format_stamp: "Formatted at 2026-09-29 01:06:26 on dist-test-slave-k0b7"
server_key: "72d86f014bc5e8712894b12946bc64f2"
server_key_iv: "ef5181120f28bebf10fcfd01955794c4"
server_key_version: "encryptionkey@0"
I20260929 01:06:26.990410 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856"
format_stamp: "Formatted at 2026-09-29 01:06:26 on dist-test-slave-k0b7"
server_key: "72d86f014bc5e8712894b12946bc64f2"
server_key_iv: "ef5181120f28bebf10fcfd01955794c4"
server_key_version: "encryptionkey@0"
I20260929 01:06:26.997231 24812 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:27.001964 24827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:27.004187 24812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:06:27.004441 24812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856"
format_stamp: "Formatted at 2026-09-29 01:06:26 on dist-test-slave-k0b7"
server_key: "72d86f014bc5e8712894b12946bc64f2"
server_key_iv: "ef5181120f28bebf10fcfd01955794c4"
server_key_version: "encryptionkey@0"
I20260929 01:06:27.004698 24812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:27.014021 24812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:27.019352 24812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:27.019639 24812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:27.039564 24812 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34055
I20260929 01:06:27.039666 24878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34055 every 8 connection(s)
I20260929 01:06:27.041114 24812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:27.044301 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24812
I20260929 01:06:27.044516 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:27.044989 3159 external_mini_cluster.cc:1428] Setting key 58f2452b61efc25b02be9b036c964ed8
I20260929 01:06:27.045491 24879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:27.053927 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:27.046965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:49129 (local address 127.0.0.1:34055)
0929 01:06:27.047807 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:27.047819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:27.047867 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:06:27.049488 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:27.049535 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:27.049912 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:27.050296 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:27.051628 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:27.051636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:27.051650 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:27.051667 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:27.051765 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:27.051789 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:27.052008 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:27.052114 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 01:06:27.052528 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":381,"threads_started":1}
I20260929 01:06:27.058650 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:27.062372 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856: Bootstrap starting.
I20260929 01:06:27.066670 24879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:27.067437 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:27.064503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:37445 (local address 127.0.0.1:34055)
0929 01:06:27.064740 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:27.064747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:27.065243 (+ 496us) server_negotiation.cc:407] Connection header received
0929 01:06:27.065397 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:27.065403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:27.065468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:27.065600 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:27.066706 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:27.066712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:27.066717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:27.066736 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:27.066788 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:27.066811 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:27.066937 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:27.067061 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 01:06:27.067136 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:06:27.068135 24879 log.cc:824] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:27.071295 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856: No bootstrap required, opened a new log
I20260929 01:06:27.079044 24879 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260929 01:06:27.079496 24879 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:27.079617 24879 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef6d00af333f4e9fbf9ab6d51b0f2856, State: Initialized, Role: FOLLOWER
I20260929 01:06:27.080286 24879 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260929 01:06:27.080549 24879 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:27.080711 24879 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:27.080897 24879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:27.083388 24879 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260929 01:06:27.083950 24879 leader_election.cc:302] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef6d00af333f4e9fbf9ab6d51b0f2856; no voters:
I20260929 01:06:27.084708 24879 leader_election.cc:288] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:27.084911 24889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:27.086097 24889 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [term 1 LEADER]: Becoming Leader. State: Replica: ef6d00af333f4e9fbf9ab6d51b0f2856, State: Running, Role: LEADER
I20260929 01:06:27.086792 24889 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260929 01:06:27.087217 24879 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:27.090596 24890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } } }
I20260929 01:06:27.090656 24891 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef6d00af333f4e9fbf9ab6d51b0f2856. Latest consensus state: current_term: 1 leader_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef6d00af333f4e9fbf9ab6d51b0f2856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } } }
I20260929 01:06:27.091149 24890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:27.091145 24891 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:27.098240 24897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:27.103144 24897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:27.114094 24897 catalog_manager.cc:1347] Generated new cluster ID: 91d92e08a8f04ed9a30b96947a535359
I20260929 01:06:27.114413 24897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:27.148034 24897 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:27.148577 24897 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:27.160118 24897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef6d00af333f4e9fbf9ab6d51b0f2856: Generated new TSK 0
I20260929 01:06:27.161916 24897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:27.181052 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24812
W20260929 01:06:27.182363 24874 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:49129
W20260929 01:06:27.183243 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24812: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:27.183411 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24812
2026-09-29T01:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2103 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:27 dist-test-slave-k0b7 krb5kdc[24910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:27 dist-test-slave-k0b7 krb5kdc[24910](info): set up 2 sockets
Sep 29 01:06:27 dist-test-slave-k0b7 krb5kdc[24910](info): commencing operation
krb5kdc: starting...
W20260929 01:06:28.343791 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:28 dist-test-slave-k0b7 krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:28.639600 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:46313
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:29.000541 24923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:29.000900 24923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:29.001087 24923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:29.009843 24923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:29.010015 24923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:29.010728 24923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:29.010989 24923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:29.011057 24923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:29.023985 24923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:46313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:29.025660 24923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:29.027165 24923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:29.040357 24928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:29.040583 24929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:29.040725 24931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:29.042248 24923 server_base.cc:1034] running on GCE node
I20260929 01:06:29.043404 24923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:29.045217 24923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:29.046663 24923 hybrid_clock.cc:648] HybridClock initialized: now 1790643989046598 us; error 54 us; skew 500 ppm
I20260929 01:06:29.047124 24923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:29 dist-test-slave-k0b7 krb5kdc[24910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643989, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:29.058815 24923 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:29.060880 24923 webserver.cc:460] Webserver started at http://127.3.21.254:43765/ using document root <none> and password file <none>
I20260929 01:06:29.061693 24923 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:29.061813 24923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:29.062204 24923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:29.065590 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "42f68184271345d5a7843592b977262a"
format_stamp: "Formatted at 2026-09-29 01:06:29 on dist-test-slave-k0b7"
server_key: "fc31be81d54a7826fa52f757c9d4678e"
server_key_iv: "43063336140a8a8ca2e7a13a01b5f553"
server_key_version: "encryptionkey@0"
I20260929 01:06:29.066439 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "42f68184271345d5a7843592b977262a"
format_stamp: "Formatted at 2026-09-29 01:06:29 on dist-test-slave-k0b7"
server_key: "fc31be81d54a7826fa52f757c9d4678e"
server_key_iv: "43063336140a8a8ca2e7a13a01b5f553"
server_key_version: "encryptionkey@0"
I20260929 01:06:29.072860 24923 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:29.077667 24938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:29.079684 24923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:06:29.079926 24923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "42f68184271345d5a7843592b977262a"
format_stamp: "Formatted at 2026-09-29 01:06:29 on dist-test-slave-k0b7"
server_key: "fc31be81d54a7826fa52f757c9d4678e"
server_key_iv: "43063336140a8a8ca2e7a13a01b5f553"
server_key_version: "encryptionkey@0"
I20260929 01:06:29.080176 24923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:29.097452 24923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:29.103858 24923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:29.104292 24923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:29.130314 24923 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:46313
I20260929 01:06:29.130335 24989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:46313 every 8 connection(s)
I20260929 01:06:29.132344 24923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:29.137235 24990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:29.138950 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 24923
I20260929 01:06:29.139171 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:29.139725 3159 external_mini_cluster.cc:1428] Setting key d61b94abff60520cd078dd7de3fe4da4
I20260929 01:06:29.152189 24990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a: Bootstrap starting.
Sep 29 01:06:29 dist-test-slave-k0b7 krb5kdc[24910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:06:29.155769 24990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:29.156926 24990 log.cc:824] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:29.160377 24990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a: No bootstrap required, opened a new log
I20260929 01:06:29.165678 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:29.141797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44323 (local address 127.3.21.254:46313)
0929 01:06:29.142790 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:29.142804 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:29.142852 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:06:29.145108 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:29.145138 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:29.145655 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:29.146206 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:29.147859 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:29.149205 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:29.150697 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:29.151456 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:29.151679 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:29.156527 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:29.156535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:29.156553 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:29.156602 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:29.160497 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:29.161599 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:29.161607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:29.161614 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:29.161805 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:29.162419 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:29.162426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:29.162429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:29.163135 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:29.163570 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:29.164104 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 01:06:29.164489 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":380,"threads_started":1}
I20260929 01:06:29.169016 24990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f68184271345d5a7843592b977262a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46313 } }
I20260929 01:06:29.169593 24990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:29.169735 24990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f68184271345d5a7843592b977262a, State: Initialized, Role: FOLLOWER
I20260929 01:06:29.170324 24990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f68184271345d5a7843592b977262a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46313 } }
I20260929 01:06:29.170632 24990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:29.170773 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:29.170796 24990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:29.170993 24990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:29.173316 24990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f68184271345d5a7843592b977262a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46313 } }
I20260929 01:06:29.173934 24990 leader_election.cc:302] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f68184271345d5a7843592b977262a; no voters:
I20260929 01:06:29.174671 24990 leader_election.cc:288] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:29.174760 24999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:29.177901 24990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:29.181561 24999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [term 1 LEADER]: Becoming Leader. State: Replica: 42f68184271345d5a7843592b977262a, State: Running, Role: LEADER
I20260929 01:06:29.182535 24999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f68184271345d5a7843592b977262a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46313 } }
I20260929 01:06:29.188344 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f68184271345d5a7843592b977262a. Latest consensus state: current_term: 1 leader_uuid: "42f68184271345d5a7843592b977262a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f68184271345d5a7843592b977262a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46313 } } }
I20260929 01:06:29.189517 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:29.189134 25000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f68184271345d5a7843592b977262a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f68184271345d5a7843592b977262a" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 46313 } } }
I20260929 01:06:29.190348 25000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:29.192849 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:29.176022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48227 (local address 127.3.21.254:46313)
0929 01:06:29.176412 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:29.176420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:29.176442 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:06:29.176502 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:29.176507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:29.176616 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:29.176706 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:29.178303 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:29.179214 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:29.180902 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:29.181388 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:29.181557 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:29.184796 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:29.184802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:29.184805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:29.184844 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:29.189242 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:29.190291 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:29.190298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:29.190301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:29.190544 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:29.191068 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:29.191073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:29.191077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:29.191397 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:29.191669 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:29.192067 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 01:06:29.192273 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260929 01:06:29.195052 25010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:29.201244 25010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:29.202760 25017 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:06:29.211792 25010 catalog_manager.cc:1347] Generated new cluster ID: 35265064ddd24927b267eb2f9527d0a7
I20260929 01:06:29.211947 25010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:29.230175 25010 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:29.232604 25010 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:29.247082 25010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f68184271345d5a7843592b977262a: Generated new TSK 0
I20260929 01:06:29.248263 25010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:29.264237 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 24923
I20260929 01:06:29.265919 24985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:29.365954 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 24923
W20260929 01:06:29.384390 24908 connection.cc:535] client connection to 127.3.21.254:46313 recv error: Network error: failed to read from TLS socket (remote: 127.3.21.254:46313): Connection reset by peer (error 104)
2026-09-29T01:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:29 dist-test-slave-k0b7 krb5kdc[25022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:29 dist-test-slave-k0b7 krb5kdc[25022](info): set up 2 sockets
Sep 29 01:06:29 dist-test-slave-k0b7 krb5kdc[25022](info): commencing operation
krb5kdc: starting...
W20260929 01:06:30.559226 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:30 dist-test-slave-k0b7 krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:30.860113 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36431
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:31.228413 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:31.228826 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:31.229010 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:31.236696 25035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:31.236902 25035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:31.237028 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:31.237142 25035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:31.237234 25035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:31.247903 25035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:36431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:36431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:31.249526 25035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:31.250986 25035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:31.264137 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:31.264206 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:31.264183 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:31.265543 25035 server_base.cc:1034] running on GCE node
I20260929 01:06:31.267290 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:31.269093 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:31.270352 25035 hybrid_clock.cc:648] HybridClock initialized: now 1790643991270274 us; error 90 us; skew 500 ppm
I20260929 01:06:31.270937 25035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:31 dist-test-slave-k0b7 krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643991, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:31.274560 25035 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:31.276325 25035 webserver.cc:460] Webserver started at http://127.3.21.254:33437/ using document root <none> and password file <none>
I20260929 01:06:31.277076 25035 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:31.277190 25035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:31.277460 25035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:31.280815 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "ac953299216e44cbb631d7d6b3387a46"
format_stamp: "Formatted at 2026-09-29 01:06:31 on dist-test-slave-k0b7"
server_key: "292cacc202006a452812d4dd0ac63c37"
server_key_iv: "3e9264e186d330f27f00b03b07a052ac"
server_key_version: "encryptionkey@0"
I20260929 01:06:31.281661 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "ac953299216e44cbb631d7d6b3387a46"
format_stamp: "Formatted at 2026-09-29 01:06:31 on dist-test-slave-k0b7"
server_key: "292cacc202006a452812d4dd0ac63c37"
server_key_iv: "3e9264e186d330f27f00b03b07a052ac"
server_key_version: "encryptionkey@0"
I20260929 01:06:31.287866 25035 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:06:31.293512 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:31.295495 25035 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:06:31.295680 25035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "ac953299216e44cbb631d7d6b3387a46"
format_stamp: "Formatted at 2026-09-29 01:06:31 on dist-test-slave-k0b7"
server_key: "292cacc202006a452812d4dd0ac63c37"
server_key_iv: "3e9264e186d330f27f00b03b07a052ac"
server_key_version: "encryptionkey@0"
I20260929 01:06:31.295879 25035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:31.322719 25035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:31.328030 25035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:31.328315 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:31.351482 25035 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:36431
I20260929 01:06:31.351517 25101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:36431 every 8 connection(s)
I20260929 01:06:31.353096 25035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:31.357461 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:31.360605 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25035
I20260929 01:06:31.360811 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:31.361362 3159 external_mini_cluster.cc:1428] Setting key 030686e8282a406f0238fef720ec161d
I20260929 01:06:31.371959 25102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46: Bootstrap starting.
I20260929 01:06:31.376143 25102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:31.377463 25102 log.cc:824] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:31.377539 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:31.363582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46673 (local address 127.3.21.254:36431)
0929 01:06:31.364597 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:31.364611 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:31.364657 (+ 46us) server_negotiation.cc:407] Connection header received
0929 01:06:31.366425 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:31.366457 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:31.367007 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:31.367365 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:31.368087 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.369814 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.371980 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.372912 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.373065 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:31.374856 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:31.374866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:31.374883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:31.375465 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:31.375569 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:31.375598 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:31.375737 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:31.376202 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 01:06:31.376420 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":482,"threads_started":1}
I20260929 01:06:31.381070 25102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46: No bootstrap required, opened a new log
I20260929 01:06:31.383285 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:31.394212 25102 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac953299216e44cbb631d7d6b3387a46" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36431 } }
I20260929 01:06:31.394836 25102 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:31.394955 25102 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac953299216e44cbb631d7d6b3387a46, State: Initialized, Role: FOLLOWER
I20260929 01:06:31.395862 25102 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac953299216e44cbb631d7d6b3387a46" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36431 } }
I20260929 01:06:31.396154 25102 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:31.396348 25102 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:31.396559 25102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:31.399632 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:31.390560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55865 (local address 127.3.21.254:36431)
0929 01:06:31.390766 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:31.390771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:31.391439 (+ 668us) server_negotiation.cc:407] Connection header received
0929 01:06:31.391706 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:31.391711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:31.391814 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:31.391964 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:31.392675 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.393644 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.395853 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.396587 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.396741 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:31.398258 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:31.398265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:31.398269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:31.398312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:31.398401 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:31.398433 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:31.398680 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:31.399227 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 01:06:31.399299 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:06:31.400019 25102 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac953299216e44cbb631d7d6b3387a46" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36431 } }
I20260929 01:06:31.400694 25102 leader_election.cc:302] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac953299216e44cbb631d7d6b3387a46; no voters:
I20260929 01:06:31.401479 25102 leader_election.cc:288] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:31.401655 25112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:31.402406 25112 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [term 1 LEADER]: Becoming Leader. State: Replica: ac953299216e44cbb631d7d6b3387a46, State: Running, Role: LEADER
I20260929 01:06:31.403127 25112 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac953299216e44cbb631d7d6b3387a46" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36431 } }
I20260929 01:06:31.403942 25102 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:31.406262 25113 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac953299216e44cbb631d7d6b3387a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac953299216e44cbb631d7d6b3387a46" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36431 } } }
I20260929 01:06:31.407403 25113 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:31.408144 25114 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac953299216e44cbb631d7d6b3387a46. Latest consensus state: current_term: 1 leader_uuid: "ac953299216e44cbb631d7d6b3387a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac953299216e44cbb631d7d6b3387a46" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 36431 } } }
I20260929 01:06:31.408638 25114 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:31.411759 25118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:31.419545 25118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:31.430722 25118 catalog_manager.cc:1347] Generated new cluster ID: 1aacb184cf124c878408d62ab8150495
I20260929 01:06:31.430905 25118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:31.501240 25118 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:31.503340 25118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:31.512228 25118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac953299216e44cbb631d7d6b3387a46: Generated new TSK 0
I20260929 01:06:31.513392 25118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:31.532792 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25035
W20260929 01:06:31.534525 25097 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46673
W20260929 01:06:31.535499 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25035: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:31.535679 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25035
W20260929 01:06:31.555773 25020 connection.cc:535] client connection to 127.3.21.254:36431 recv error: Network error: failed to read from TLS socket (remote: 127.3.21.254:36431): Connection reset by peer (error 104)
2026-09-29T01:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2171 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:31 dist-test-slave-k0b7 krb5kdc[25133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:31 dist-test-slave-k0b7 krb5kdc[25133](info): set up 2 sockets
Sep 29 01:06:31 dist-test-slave-k0b7 krb5kdc[25133](info): commencing operation
krb5kdc: starting...
W20260929 01:06:32.703475 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:32 dist-test-slave-k0b7 krb5kdc[25133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:33.009577 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:38589
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:38589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:33.403805 25146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:33.404137 25146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:33.404304 25146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:33.412073 25146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:33.412216 25146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:33.412305 25146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:33.412389 25146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:33.412449 25146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:33.423138 25146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:38589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:33.424964 25146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:33.426426 25146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:33.440160 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:33.440210 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:33.440186 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:33.441335 25146 server_base.cc:1034] running on GCE node
I20260929 01:06:33.442180 25146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:33.443856 25146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:33.445134 25146 hybrid_clock.cc:648] HybridClock initialized: now 1790643993445077 us; error 57 us; skew 500 ppm
I20260929 01:06:33.445477 25146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:33 dist-test-slave-k0b7 krb5kdc[25133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643993, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:33.449534 25146 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:33.451208 25146 webserver.cc:460] Webserver started at http://127.3.21.254:38103/ using document root <none> and password file <none>
I20260929 01:06:33.451910 25146 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:33.452023 25146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:33.452279 25146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:33.455861 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "e460badb2d554d299b4db17d0e2cd5d3"
format_stamp: "Formatted at 2026-09-29 01:06:33 on dist-test-slave-k0b7"
server_key: "4d7aba759806599a3b73b2d12b0346bb"
server_key_iv: "2870e0c473dbe60da095155abc05fd9e"
server_key_version: "encryptionkey@0"
I20260929 01:06:33.456854 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "e460badb2d554d299b4db17d0e2cd5d3"
format_stamp: "Formatted at 2026-09-29 01:06:33 on dist-test-slave-k0b7"
server_key: "4d7aba759806599a3b73b2d12b0346bb"
server_key_iv: "2870e0c473dbe60da095155abc05fd9e"
server_key_version: "encryptionkey@0"
I20260929 01:06:33.463819 25146 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:06:33.467932 25161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:33.469938 25146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:06:33.470181 25146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "e460badb2d554d299b4db17d0e2cd5d3"
format_stamp: "Formatted at 2026-09-29 01:06:33 on dist-test-slave-k0b7"
server_key: "4d7aba759806599a3b73b2d12b0346bb"
server_key_iv: "2870e0c473dbe60da095155abc05fd9e"
server_key_version: "encryptionkey@0"
I20260929 01:06:33.470453 25146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:33.478072 25146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:33.483366 25146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:33.483705 25146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:33.505640 25146 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:38589
I20260929 01:06:33.505703 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:38589 every 8 connection(s)
I20260929 01:06:33.507195 25146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:33.511169 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25146
I20260929 01:06:33.511380 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:33.511861 25213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:33.511896 3159 external_mini_cluster.cc:1428] Setting key 6750905fb22c73b0115998fb01296c91
I20260929 01:06:33.522135 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:33.514019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39593 (local address 127.3.21.254:38589)
0929 01:06:33.514974 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:33.515003 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:33.515055 (+ 52us) server_negotiation.cc:407] Connection header received
0929 01:06:33.517461 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:33.517496 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:33.518185 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:33.518752 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:33.519847 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:33.519861 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:33.519879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:33.519899 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:33.519975 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:33.520024 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:33.520267 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:33.520419 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 01:06:33.520647 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":406,"threads_started":1}
I20260929 01:06:33.527068 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:33.528929 25213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3: Bootstrap starting.
I20260929 01:06:33.533102 25213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:33.534294 25213 log.cc:824] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:33.534896 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:33.532016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34995 (local address 127.3.21.254:38589)
0929 01:06:33.532243 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:33.532251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:33.532535 (+ 284us) server_negotiation.cc:407] Connection header received
0929 01:06:33.532761 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:33.532766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:33.532820 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:33.532922 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:33.534107 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:33.534117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:33.534122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:33.534147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:33.534192 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:33.534209 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:33.534354 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:33.534514 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 01:06:33.534589 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:06:33.537667 25213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3: No bootstrap required, opened a new log
I20260929 01:06:33.545994 25213 raft_consensus.cc:348] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e460badb2d554d299b4db17d0e2cd5d3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38589 } }
I20260929 01:06:33.546430 25213 raft_consensus.cc:374] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:33.546562 25213 raft_consensus.cc:729] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e460badb2d554d299b4db17d0e2cd5d3, State: Initialized, Role: FOLLOWER
I20260929 01:06:33.547175 25213 consensus_queue.cc:260] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e460badb2d554d299b4db17d0e2cd5d3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38589 } }
I20260929 01:06:33.547400 25213 raft_consensus.cc:388] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:33.547511 25213 raft_consensus.cc:482] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:33.547675 25213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:33.550110 25213 raft_consensus.cc:504] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e460badb2d554d299b4db17d0e2cd5d3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38589 } }
I20260929 01:06:33.550770 25213 leader_election.cc:302] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e460badb2d554d299b4db17d0e2cd5d3; no voters:
I20260929 01:06:33.551636 25213 leader_election.cc:288] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:33.551743 25223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:33.552407 25223 raft_consensus.cc:686] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [term 1 LEADER]: Becoming Leader. State: Replica: e460badb2d554d299b4db17d0e2cd5d3, State: Running, Role: LEADER
I20260929 01:06:33.553088 25223 consensus_queue.cc:237] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e460badb2d554d299b4db17d0e2cd5d3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38589 } }
I20260929 01:06:33.554167 25213 sys_catalog.cc:564] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:33.556927 25225 sys_catalog.cc:455] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e460badb2d554d299b4db17d0e2cd5d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e460badb2d554d299b4db17d0e2cd5d3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38589 } } }
I20260929 01:06:33.557406 25225 sys_catalog.cc:458] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:33.557118 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e460badb2d554d299b4db17d0e2cd5d3. Latest consensus state: current_term: 1 leader_uuid: "e460badb2d554d299b4db17d0e2cd5d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e460badb2d554d299b4db17d0e2cd5d3" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 38589 } } }
I20260929 01:06:33.557539 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:33.564271 25231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:33.569665 25231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:33.579839 25231 catalog_manager.cc:1347] Generated new cluster ID: 5a589a93fbbe4eccb055b7e4c5e5cd0f
I20260929 01:06:33.579991 25231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:33.591550 25231 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:33.592166 25231 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:33.603143 25231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e460badb2d554d299b4db17d0e2cd5d3: Generated new TSK 0
I20260929 01:06:33.604506 25231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:33.614032 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25146
W20260929 01:06:33.615504 25208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39593
W20260929 01:06:33.616386 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25146: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:33.616557 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25146
2026-09-29T01:06:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2079 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:33 dist-test-slave-k0b7 krb5kdc[25244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:33 dist-test-slave-k0b7 krb5kdc[25244](info): set up 2 sockets
Sep 29 01:06:33 dist-test-slave-k0b7 krb5kdc[25244](info): commencing operation
krb5kdc: starting...
W20260929 01:06:34.823762 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:35 dist-test-slave-k0b7 krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:35.138856 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34809
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:39243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:35.503106 25257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:35.503424 25257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:35.503593 25257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:35.511508 25257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:35.511641 25257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:35.511732 25257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:35.511822 25257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:35.511904 25257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:35.522384 25257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:39243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:35.524075 25257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:35.525491 25257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:35.540261 25262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:35.540364 25265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:35.540359 25263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:35.541538 25257 server_base.cc:1034] running on GCE node
I20260929 01:06:35.542680 25257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:35.544513 25257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:35.545832 25257 hybrid_clock.cc:648] HybridClock initialized: now 1790643995545786 us; error 73 us; skew 500 ppm
I20260929 01:06:35.546288 25257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:35 dist-test-slave-k0b7 krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:35.558684 25257 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:35.560628 25257 webserver.cc:460] Webserver started at http://127.3.21.254:41695/ using document root <none> and password file <none>
I20260929 01:06:35.561441 25257 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:35.561617 25257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:35.561988 25257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:35.565354 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "545b921f82ca491295b751577a2e6784"
format_stamp: "Formatted at 2026-09-29 01:06:35 on dist-test-slave-k0b7"
server_key: "e31abfcb0d0dfb75b7a5d63ea79a971b"
server_key_iv: "70aa70fb652b80393d725aea5c634b53"
server_key_version: "encryptionkey@0"
I20260929 01:06:35.566224 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "545b921f82ca491295b751577a2e6784"
format_stamp: "Formatted at 2026-09-29 01:06:35 on dist-test-slave-k0b7"
server_key: "e31abfcb0d0dfb75b7a5d63ea79a971b"
server_key_iv: "70aa70fb652b80393d725aea5c634b53"
server_key_version: "encryptionkey@0"
I20260929 01:06:35.572708 25257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:35.577049 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:35.579133 25257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:06:35.579393 25257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "545b921f82ca491295b751577a2e6784"
format_stamp: "Formatted at 2026-09-29 01:06:35 on dist-test-slave-k0b7"
server_key: "e31abfcb0d0dfb75b7a5d63ea79a971b"
server_key_iv: "70aa70fb652b80393d725aea5c634b53"
server_key_version: "encryptionkey@0"
I20260929 01:06:35.579653 25257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:35.597108 25257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:35.602115 25257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:35.602512 25257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:35.624197 25257 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34809
I20260929 01:06:35.624238 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34809 every 8 connection(s)
I20260929 01:06:35.625712 25257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:35.629045 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25257
I20260929 01:06:35.629266 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:35.629837 3159 external_mini_cluster.cc:1428] Setting key c93095e12727d15f9d8ffc148db0bd31
I20260929 01:06:35.630622 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:06:35 dist-test-slave-k0b7 krb5kdc[25244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:06:35.645167 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784: Bootstrap starting.
I20260929 01:06:35.648648 25324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:35.649798 25324 log.cc:824] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:35.652974 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784: No bootstrap required, opened a new log
I20260929 01:06:35.653242 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:35.631998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59251 (local address 127.3.21.254:34809)
0929 01:06:35.632844 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:35.632855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:35.632897 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:06:35.634631 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:35.634655 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:35.635151 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:35.635595 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:35.637127 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.638630 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.640023 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.640741 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.640948 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:35.645456 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:35.645469 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:35.645492 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:35.645537 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:35.648752 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.649593 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.649603 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.649616 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.649805 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.650240 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.650246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.650249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.650915 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:35.651242 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:35.651703 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 01:06:35.652155 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":380,"threads_started":1}
I20260929 01:06:35.657267 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:35.661757 25324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545b921f82ca491295b751577a2e6784" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34809 } }
I20260929 01:06:35.662331 25324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:35.662452 25324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545b921f82ca491295b751577a2e6784, State: Initialized, Role: FOLLOWER
I20260929 01:06:35.663260 25324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545b921f82ca491295b751577a2e6784" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34809 } }
I20260929 01:06:35.663568 25324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:35.663744 25324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:35.663935 25324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:35.667213 25324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545b921f82ca491295b751577a2e6784" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34809 } }
I20260929 01:06:35.667843 25324 leader_election.cc:302] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545b921f82ca491295b751577a2e6784; no voters:
I20260929 01:06:35.668705 25324 leader_election.cc:288] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:35.668871 25334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:35.670977 25334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [term 1 LEADER]: Becoming Leader. State: Replica: 545b921f82ca491295b751577a2e6784, State: Running, Role: LEADER
I20260929 01:06:35.671439 25324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:35.671679 25334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545b921f82ca491295b751577a2e6784" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34809 } }
I20260929 01:06:35.676753 25335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545b921f82ca491295b751577a2e6784" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545b921f82ca491295b751577a2e6784" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34809 } } }
I20260929 01:06:35.677242 25335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:35.677749 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545b921f82ca491295b751577a2e6784. Latest consensus state: current_term: 1 leader_uuid: "545b921f82ca491295b751577a2e6784" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545b921f82ca491295b751577a2e6784" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34809 } } }
I20260929 01:06:35.678056 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:35.682791 25341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:35.683787 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:35.662218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46363 (local address 127.3.21.254:34809)
0929 01:06:35.662429 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:35.662438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:35.663127 (+ 689us) server_negotiation.cc:407] Connection header received
0929 01:06:35.663282 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:35.663289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:35.663375 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:35.663489 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:35.665071 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.666275 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.667937 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.668792 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.668945 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:35.672747 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:35.672754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:35.672758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:35.672816 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:35.677995 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.679174 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.679182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.679187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.679430 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.680050 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.680059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.680065 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.680634 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:35.681063 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:35.682612 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0929 01:06:35.683049 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:06:35.690146 25341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:35.700044 25341 catalog_manager.cc:1347] Generated new cluster ID: da9e34679c4e4eb9a13e8deb13f0612d
I20260929 01:06:35.700218 25341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:35.708990 25341 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:35.710831 25341 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:35.730109 25341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 545b921f82ca491295b751577a2e6784: Generated new TSK 0
I20260929 01:06:35.731176 25341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:35.743522 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25257
I20260929 01:06:35.745553 25319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:35.843252 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25257
2026-09-29T01:06:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:35 dist-test-slave-k0b7 krb5kdc[25356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:35 dist-test-slave-k0b7 krb5kdc[25356](info): set up 2 sockets
Sep 29 01:06:35 dist-test-slave-k0b7 krb5kdc[25356](info): commencing operation
krb5kdc: starting...
W20260929 01:06:37.011422 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:37 dist-test-slave-k0b7 krb5kdc[25356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:37.312348 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34383
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:37.686435 25369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:37.686846 25369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:37.687024 25369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:37.694697 25369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:37.694831 25369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:37.694918 25369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:37.695034 25369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:37.695099 25369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:37.706375 25369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:37.707901 25369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:37.709394 25369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:37.723770 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:37.723834 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:37.723802 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:37.724934 25369 server_base.cc:1034] running on GCE node
I20260929 01:06:37.725936 25369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:37.727859 25369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:37.729157 25369 hybrid_clock.cc:648] HybridClock initialized: now 1790643997729103 us; error 69 us; skew 500 ppm
I20260929 01:06:37.729820 25369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:37 dist-test-slave-k0b7 krb5kdc[25356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:37.733618 25369 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:37.735334 25369 webserver.cc:460] Webserver started at http://127.3.21.254:37621/ using document root <none> and password file <none>
I20260929 01:06:37.736047 25369 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:37.736152 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:37.736408 25369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:37.739708 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "3fdc1f5481534e229cfa8ddf8873d91d"
format_stamp: "Formatted at 2026-09-29 01:06:37 on dist-test-slave-k0b7"
server_key: "dab4b4744025f0425af6fdf58640c28e"
server_key_iv: "16ab6ce87cef336ac80e900f4e786368"
server_key_version: "encryptionkey@0"
I20260929 01:06:37.740474 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "3fdc1f5481534e229cfa8ddf8873d91d"
format_stamp: "Formatted at 2026-09-29 01:06:37 on dist-test-slave-k0b7"
server_key: "dab4b4744025f0425af6fdf58640c28e"
server_key_iv: "16ab6ce87cef336ac80e900f4e786368"
server_key_version: "encryptionkey@0"
I20260929 01:06:37.746297 25369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:06:37.750767 25384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:37.752699 25369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:06:37.752889 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "3fdc1f5481534e229cfa8ddf8873d91d"
format_stamp: "Formatted at 2026-09-29 01:06:37 on dist-test-slave-k0b7"
server_key: "dab4b4744025f0425af6fdf58640c28e"
server_key_iv: "16ab6ce87cef336ac80e900f4e786368"
server_key_version: "encryptionkey@0"
I20260929 01:06:37.753122 25369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:37.771240 25369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:37.776376 25369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:37.776669 25369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:37.799129 25369 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34383
I20260929 01:06:37.799168 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34383 every 8 connection(s)
I20260929 01:06:37.800724 25369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:37.805709 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:37.808861 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25369
I20260929 01:06:37.809080 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:37.809648 3159 external_mini_cluster.cc:1428] Setting key f09e9e5e6a0fda6870dcd7dfac6ae8a4
I20260929 01:06:37.821031 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d: Bootstrap starting.
I20260929 01:06:37.825452 25436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:37.826894 25436 log.cc:824] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:37.827884 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:37.812183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36451 (local address 127.3.21.254:34383)
0929 01:06:37.813121 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:37.813139 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:37.813193 (+ 54us) server_negotiation.cc:407] Connection header received
0929 01:06:37.815683 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:37.815718 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:37.816364 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:37.816906 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:37.817517 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:37.819718 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:37.821365 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:37.822471 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:37.822727 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:37.824356 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:37.824367 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:37.824389 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:37.825201 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:37.825321 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:37.825361 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:37.825522 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:37.825995 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 01:06:37.826324 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":404,"threads_started":1}
I20260929 01:06:37.830760 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d: No bootstrap required, opened a new log
I20260929 01:06:37.832934 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:37.839704 25436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34383 } }
I20260929 01:06:37.840255 25436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:37.840381 25436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdc1f5481534e229cfa8ddf8873d91d, State: Initialized, Role: FOLLOWER
I20260929 01:06:37.841099 25436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34383 } }
I20260929 01:06:37.841387 25436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:37.841537 25436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:37.841734 25436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:37.844936 25436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34383 } }
I20260929 01:06:37.845455 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:37.837755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54977 (local address 127.3.21.254:34383)
0929 01:06:37.837975 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:37.837984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:37.838395 (+ 411us) server_negotiation.cc:407] Connection header received
0929 01:06:37.838688 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:37.838696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:37.838767 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:37.838884 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:37.839430 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:37.840207 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:37.842085 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:37.842768 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:37.842958 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:37.844411 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:37.844417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:37.844420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:37.844443 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:37.844492 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:37.844515 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:37.844645 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:37.844813 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 01:06:37.844925 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:06:37.845889 25436 leader_election.cc:302] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fdc1f5481534e229cfa8ddf8873d91d; no voters:
I20260929 01:06:37.846626 25436 leader_election.cc:288] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:37.846874 25446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:37.847610 25446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [term 1 LEADER]: Becoming Leader. State: Replica: 3fdc1f5481534e229cfa8ddf8873d91d, State: Running, Role: LEADER
I20260929 01:06:37.848299 25446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34383 } }
I20260929 01:06:37.849530 25436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:37.851227 25448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34383 } } }
I20260929 01:06:37.851742 25448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:37.851609 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fdc1f5481534e229cfa8ddf8873d91d. Latest consensus state: current_term: 1 leader_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc1f5481534e229cfa8ddf8873d91d" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34383 } } }
I20260929 01:06:37.852349 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:37.853413 25453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:37.863202 25453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:37.875650 25453 catalog_manager.cc:1347] Generated new cluster ID: a1b1404ade5d42fbb347036d132d6dbd
I20260929 01:06:37.875794 25453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:37.890115 25453 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:37.891815 25453 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:37.903210 25453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fdc1f5481534e229cfa8ddf8873d91d: Generated new TSK 0
I20260929 01:06:37.904505 25453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:37.915561 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25369
W20260929 01:06:37.917506 25431 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36451
W20260929 01:06:37.918402 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25369: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:37.918601 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25369
2026-09-29T01:06:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:38 dist-test-slave-k0b7 krb5kdc[25467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:38 dist-test-slave-k0b7 krb5kdc[25467](info): set up 2 sockets
Sep 29 01:06:38 dist-test-slave-k0b7 krb5kdc[25467](info): commencing operation
krb5kdc: starting...
W20260929 01:06:39.107426 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:39 dist-test-slave-k0b7 krb5kdc[25467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:39.416535 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:33433
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:33433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:39.805521 25480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:39.805876 25480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:39.805984 25480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:39.814637 25480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:39.814850 25480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:39.814975 25480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:39.815101 25480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:39.815200 25480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:39.828047 25480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:33433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:33433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:39.829697 25480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:39.831290 25480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:39.846455 25485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.846540 25488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.846454 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:39.847520 25480 server_base.cc:1034] running on GCE node
I20260929 01:06:39.848450 25480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:39.850140 25480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:39.851414 25480 hybrid_clock.cc:648] HybridClock initialized: now 1790643999851380 us; error 38 us; skew 500 ppm
I20260929 01:06:39.851958 25480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:39 dist-test-slave-k0b7 krb5kdc[25467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:39.855898 25480 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:39.857815 25480 webserver.cc:460] Webserver started at http://127.3.21.254:35499/ using document root <none> and password file <none>
I20260929 01:06:39.858752 25480 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:39.858884 25480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:39.859171 25480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:39.862795 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "7fe2c59320d94d22a290d0649e66c844"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-k0b7"
server_key: "523cf661ed1bcccad8191801f1b91e75"
server_key_iv: "6b82f3a75bc14a1ce9c761e2095d4675"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.863709 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "7fe2c59320d94d22a290d0649e66c844"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-k0b7"
server_key: "523cf661ed1bcccad8191801f1b91e75"
server_key_iv: "6b82f3a75bc14a1ce9c761e2095d4675"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.870224 25480 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 01:06:39.875205 25495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:39.877039 25480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:06:39.877313 25480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "7fe2c59320d94d22a290d0649e66c844"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-k0b7"
server_key: "523cf661ed1bcccad8191801f1b91e75"
server_key_iv: "6b82f3a75bc14a1ce9c761e2095d4675"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.877559 25480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:39.884781 25480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:39.889806 25480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:39.890161 25480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:39.912631 25480 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:33433
I20260929 01:06:39.912743 25546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:33433 every 8 connection(s)
I20260929 01:06:39.914178 25480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:39.917520 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25480
I20260929 01:06:39.917707 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:39.918248 3159 external_mini_cluster.cc:1428] Setting key 7816dc4bc731e6e0f233322bdb93345f
I20260929 01:06:39.919013 25547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:39.930063 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.920479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44861 (local address 127.3.21.254:33433)
0929 01:06:39.921575 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:39.921588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:39.921641 (+ 53us) server_negotiation.cc:407] Connection header received
0929 01:06:39.924587 (+ 2946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:39.924619 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:39.925084 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:39.925499 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:39.927794 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:39.927809 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:39.927832 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:39.927859 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:39.927947 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:39.927979 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:39.928233 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:39.928370 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 01:06:39.928833 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":482,"threads_started":1}
I20260929 01:06:39.936546 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:39.937918 25547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844: Bootstrap starting.
I20260929 01:06:39.943110 25547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:39.944986 25547 log.cc:824] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:39.946607 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.942905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49971 (local address 127.3.21.254:33433)
0929 01:06:39.943154 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:39.943162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:39.943823 (+ 661us) server_negotiation.cc:407] Connection header received
0929 01:06:39.944009 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:39.944014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:39.944085 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:39.944220 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:39.945523 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:39.945530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:39.945535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:39.945557 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:39.945612 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:39.945636 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:39.945797 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:39.946008 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 01:06:39.946087 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 01:06:39.949476 25547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844: No bootstrap required, opened a new log
I20260929 01:06:39.958649 25547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2c59320d94d22a290d0649e66c844" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33433 } }
I20260929 01:06:39.959101 25547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:39.959276 25547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe2c59320d94d22a290d0649e66c844, State: Initialized, Role: FOLLOWER
I20260929 01:06:39.959894 25547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2c59320d94d22a290d0649e66c844" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33433 } }
I20260929 01:06:39.960152 25547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:39.960307 25547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:39.960489 25547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:39.962901 25547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2c59320d94d22a290d0649e66c844" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33433 } }
I20260929 01:06:39.963469 25547 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe2c59320d94d22a290d0649e66c844; no voters:
I20260929 01:06:39.964165 25547 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:39.964359 25557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:39.965862 25557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe2c59320d94d22a290d0649e66c844, State: Running, Role: LEADER
I20260929 01:06:39.966650 25547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:39.966574 25557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2c59320d94d22a290d0649e66c844" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33433 } }
I20260929 01:06:39.970057 25559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe2c59320d94d22a290d0649e66c844. Latest consensus state: current_term: 1 leader_uuid: "7fe2c59320d94d22a290d0649e66c844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2c59320d94d22a290d0649e66c844" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33433 } } }
I20260929 01:06:39.970602 25559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:39.970085 25558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe2c59320d94d22a290d0649e66c844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2c59320d94d22a290d0649e66c844" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 33433 } } }
I20260929 01:06:39.971664 25558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:39.973235 25567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:39.977957 25567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:39.987399 25567 catalog_manager.cc:1347] Generated new cluster ID: aab0f3b379da4c9ea0497341a07b199a
I20260929 01:06:39.987526 25567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:40.012818 25567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:40.013506 25567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:40.030958 25567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe2c59320d94d22a290d0649e66c844: Generated new TSK 0
I20260929 01:06:40.031941 25567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:40.052383 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25480
W20260929 01:06:40.053853 25542 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44861
W20260929 01:06:40.054927 3159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 01:06:40.055146 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25480
2026-09-29T01:06:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2136 ms)
[----------] 18 tests from AuthTokenIssuingTest (38331 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:40 dist-test-slave-k0b7 krb5kdc[25578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:40 dist-test-slave-k0b7 krb5kdc[25578](info): set up 2 sockets
Sep 29 01:06:40 dist-test-slave-k0b7 krb5kdc[25578](info): commencing operation
krb5kdc: starting...
W20260929 01:06:41.252074 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:41 dist-test-slave-k0b7 krb5kdc[25578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:41.564464 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34131
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34131,127.3.21.253:39747,127.3.21.252:34463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:41.933384 25591 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:41.933828 25591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:41.933947 25591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:41.934100 25591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:41.943099 25591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:41.943219 25591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:41.943317 25591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:41.943403 25591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:41.943500 25591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:41.954304 25591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34131,127.3.21.253:39747,127.3.21.252:34463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:41.955824 25591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:41.957309 25591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:41.971436 25597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:41.971647 25599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:41.971558 25596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:41.973104 25591 server_base.cc:1034] running on GCE node
I20260929 01:06:41.974140 25591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:41.975811 25591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:41.977124 25591 hybrid_clock.cc:648] HybridClock initialized: now 1790644001977046 us; error 47 us; skew 500 ppm
I20260929 01:06:41.977519 25591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:41 dist-test-slave-k0b7 krb5kdc[25578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:41.989851 25591 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:41.991829 25591 webserver.cc:460] Webserver started at http://127.3.21.254:40227/ using document root <none> and password file <none>
I20260929 01:06:41.992624 25591 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:41.992776 25591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:41.993161 25591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:41.996446 25591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "363fa8fe58624fcaa8a50985cc8502c1"
format_stamp: "Formatted at 2026-09-29 01:06:41 on dist-test-slave-k0b7"
server_key: "2808d4a48d25e2c4e1fdfc7f35023ae4"
server_key_iv: "5fe8383c8567b6bac454718e6dcbaca0"
server_key_version: "encryptionkey@0"
I20260929 01:06:41.997264 25591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "363fa8fe58624fcaa8a50985cc8502c1"
format_stamp: "Formatted at 2026-09-29 01:06:41 on dist-test-slave-k0b7"
server_key: "2808d4a48d25e2c4e1fdfc7f35023ae4"
server_key_iv: "5fe8383c8567b6bac454718e6dcbaca0"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.003171 25591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 01:06:42.008095 25606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:42.009912 25591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:06:42.010104 25591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "363fa8fe58624fcaa8a50985cc8502c1"
format_stamp: "Formatted at 2026-09-29 01:06:41 on dist-test-slave-k0b7"
server_key: "2808d4a48d25e2c4e1fdfc7f35023ae4"
server_key_iv: "5fe8383c8567b6bac454718e6dcbaca0"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.010335 25591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:42.025761 25591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:42.031430 25591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:42.031710 25591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:42.054157 25591 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:34131
I20260929 01:06:42.054211 25657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:34131 every 8 connection(s)
I20260929 01:06:42.055807 25591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:42.060338 25658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:42.065137 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25591
I20260929 01:06:42.065407 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:42.066231 3159 external_mini_cluster.cc:1428] Setting key 0222fe8ea70fc8eecbd7d6551f2810ce
I20260929 01:06:42.068607 25658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:42 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.253@KRBTEST.COM; defaulting to no policy
I20260929 01:06:42.094904 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.072529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.3.21.254:34131)
0929 01:06:42.073606 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.073629 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.074125 (+ 496us) server_negotiation.cc:407] Connection header received
0929 01:06:42.076530 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.076537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.076814 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.077060 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.078658 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.079987 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.081564 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.082234 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.082478 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.086619 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.086632 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.086652 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.086721 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.090096 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.090988 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.090995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.090997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.091153 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.091659 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.091665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.091669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.092284 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.092471 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.093265 (+ 794us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.093866 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":239,"server-negotiator.queue_time_us":779,"thread_start_us":508,"threads_started":1}
I20260929 01:06:42.095047 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.072530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34131 (local address 127.0.0.1:33182)
0929 01:06:42.073928 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.073971 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.076091 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.077080 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.077088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.078019 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.078387 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.078407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.080185 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.080194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.081399 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.081413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.082545 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.082554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.082757 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.083530 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.083556 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.086299 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.090260 (+ 3961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.090274 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.090287 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.090834 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.091284 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.091291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.091294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.091463 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.092449 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.092456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.092836 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.093054 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.093496 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":906,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.253@KRBTEST.COM" created.
I20260929 01:06:42.098862 25658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.100306 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.099845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42222)
0929 01:06:42.100053 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.100179 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 01:06:42.100643 25608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
I20260929 01:06:42.102394 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.102014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42226)
0929 01:06:42.102177 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.102255 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:06:42.102659 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:39747: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab.
Entry for principal kudu/127.3.21.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab.
I20260929 01:06:42.131785 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:39747
--webserver_interface=127.3.21.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab
--principal=kudu/127.3.21.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34131,127.3.21.253:39747,127.3.21.252:34463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:06:42.152037 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } attempt: 1
I20260929 01:06:42.153393 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.152893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42232)
0929 01:06:42.153148 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.153241 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 01:06:42.154819 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.154381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42238)
0929 01:06:42.154600 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.154676 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260929 01:06:42.155066 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:39747: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
I20260929 01:06:42.223402 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } attempt: 2
I20260929 01:06:42.224641 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.224127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42244)
0929 01:06:42.224398 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.224499 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 01:06:42.226079 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.225438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42260)
0929 01:06:42.225753 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.225834 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 01:06:42.226291 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:39747: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
I20260929 01:06:42.317683 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } attempt: 3
I20260929 01:06:42.318996 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.318387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42262)
0929 01:06:42.318680 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.318817 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 01:06:42.320439 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.319940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42270)
0929 01:06:42.320164 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.320253 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 01:06:42.320731 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:39747: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
I20260929 01:06:42.464154 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } attempt: 4
I20260929 01:06:42.465675 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.464991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42276)
0929 01:06:42.465344 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.465460 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260929 01:06:42.467311 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.466784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42282)
0929 01:06:42.466975 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.467143 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 01:06:42.467736 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:39747: Network error: Client connection negotiation failed: client connection to 127.3.21.253:39747: connect: Connection refused (error 111)
W20260929 01:06:42.514564 25665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:42.515076 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:42.515194 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:42.515352 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:42.523327 25665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:42.523691 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:42.523851 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:42.523957 25665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:42.524052 25665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:42.535871 25665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34131,127.3.21.253:39747,127.3.21.252:34463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab
--principal=kudu/127.3.21.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.21.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:42.537420 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:42.538940 25665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:42.555261 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:42.555302 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:42.555341 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:42.555261 25665 server_base.cc:1034] running on GCE node
I20260929 01:06:42.556684 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:42.558770 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:42.560228 25665 hybrid_clock.cc:648] HybridClock initialized: now 1790644002560075 us; error 118 us; skew 500 ppm
I20260929 01:06:42.560812 25665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:42 dist-test-slave-k0b7 krb5kdc[25578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:42.564831 25665 init.cc:377] Logged in from keytab as kudu/127.3.21.253@KRBTEST.COM (short username kudu)
I20260929 01:06:42.566844 25665 webserver.cc:460] Webserver started at http://127.3.21.253:36293/ using document root <none> and password file <none>
I20260929 01:06:42.567689 25665 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:42.567847 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:42.568231 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:42.571697 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data/instance:
uuid: "54395886af59486b81f24c4cec330c83"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-k0b7"
server_key: "a8f63e2e40f16ceab9b3c96810ac1f0d"
server_key_iv: "dd0f78df23ceace265a5622dceb494a9"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.572618 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal/instance:
uuid: "54395886af59486b81f24c4cec330c83"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-k0b7"
server_key: "a8f63e2e40f16ceab9b3c96810ac1f0d"
server_key_iv: "dd0f78df23ceace265a5622dceb494a9"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.579005 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 01:06:42.583231 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:42.585134 25665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:06:42.585462 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal
uuid: "54395886af59486b81f24c4cec330c83"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-k0b7"
server_key: "a8f63e2e40f16ceab9b3c96810ac1f0d"
server_key_iv: "dd0f78df23ceace265a5622dceb494a9"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.585723 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:42.599316 25665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:42.604797 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:42.605127 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:42.629273 25665 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.253:39747
I20260929 01:06:42.629316 25731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.253:39747 every 8 connection(s)
I20260929 01:06:42.630860 25665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
I20260929 01:06:42.633181 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25665
I20260929 01:06:42.633409 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-1/wal/instance
I20260929 01:06:42.633955 3159 external_mini_cluster.cc:1428] Setting key 82dc14046adb46c09399e3423a863527
I20260929 01:06:42.637133 25732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:42.648300 25732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.21.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.252@KRBTEST.COM" created.
Sep 29 01:06:42 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:06:42.677233 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.651402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.3.21.254:34131)
0929 01:06:42.652097 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.652107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.653187 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 01:06:42.655483 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.655492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.655607 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.655742 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.657789 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.659148 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.661024 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.661794 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.661957 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.668001 (+ 6044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.668013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.668021 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.668087 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.672679 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.673849 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.673860 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.673867 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.674058 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.674833 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.674843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.674848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.675286 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.675471 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.676478 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.676785 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":301,"threads_started":1}
I20260929 01:06:42.678759 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.652059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34131 (local address 127.0.0.1:33184)
0929 01:06:42.652969 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.653031 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.655150 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.656077 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.656087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.656817 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.657511 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.657532 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.659379 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.659384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.660825 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.660840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.662092 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.662099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.662416 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.663619 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.663645 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.667647 (+ 4002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.672951 (+ 5304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.672960 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.672977 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.673550 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.674225 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.674229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.674231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.674662 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.675477 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.675484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.676029 (+ 545us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.676303 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.676778 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":416,"threads_started":1}
I20260929 01:06:42.680958 25732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:42 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:06:42.699961 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.682837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42294)
0929 01:06:42.683055 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.683088 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.683216 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.684336 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.684341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.684825 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.684967 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.684976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.686399 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.686405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.687338 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.687350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.688226 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.688231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.688443 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.689101 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.689159 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.692063 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.696125 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.696129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.696132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.696633 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.697312 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.697316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.697319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.697460 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.698546 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.698554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.699106 (+ 552us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.699240 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.699472 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":4}
I20260929 01:06:42.700706 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.682981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.3.21.253:39747)
0929 01:06:42.683846 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.683853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.683881 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:06:42.683935 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.683941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.684144 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.684357 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.685151 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.686249 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.687498 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.688068 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.688215 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.692345 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.692358 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.692374 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.692408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.695871 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.696826 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.696834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.696836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.697095 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.697797 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.697801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.697803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.698319 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.698601 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.699931 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.700252 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":696,"thread_start_us":486,"threads_started":1}
I20260929 01:06:42.703707 25732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.704957 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.704469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:38976)
0929 01:06:42.704670 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.704802 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 01:06:42.705266 25684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab.
Entry for principal kudu/127.3.21.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab.
I20260929 01:06:42.707010 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.706544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:38978)
0929 01:06:42.706799 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.706899 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260929 01:06:42.707873 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:34463
--webserver_interface=127.3.21.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab
--principal=kudu/127.3.21.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:34131,127.3.21.253:39747,127.3.21.252:34463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:42.708401 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:42.757938 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 1
I20260929 01:06:42.759416 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.758865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:38990)
0929 01:06:42.759091 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.759222 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 01:06:42.760655 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.760226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:38996)
0929 01:06:42.760408 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.760500 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 01:06:42.760957 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:42.767004 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } attempt: 5
Sep 29 01:06:42 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:06:42.783242 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.768265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42296)
0929 01:06:42.768571 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.768593 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.768735 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.769731 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.769737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.770198 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.770379 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.770389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.771511 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.771517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.772438 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.772452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.773242 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.773247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.773454 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.774170 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.774189 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.776629 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.780338 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.780350 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.780354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.780793 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.781248 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.781256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.781260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.781416 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.782051 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.782057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.782251 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.782455 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.782827 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260929 01:06:42.783242 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.768401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.3.21.253:39747)
0929 01:06:42.769314 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.769320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.769340 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:06:42.769410 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.769416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.769524 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.769669 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.770600 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.771324 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.772625 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.773058 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.773197 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.776869 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.776874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.776878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.776915 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.780112 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.780956 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.780963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.780967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.781110 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.781549 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.781555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.781558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.781909 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.782093 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.782599 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.782858 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776}
I20260929 01:06:42.785022 25658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.786324 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.785770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39006)
0929 01:06:42.785984 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.786087 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 01:06:42.787714 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.787292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39010)
0929 01:06:42.787461 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.787560 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 01:06:42.787972 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:42.829259 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 2
I20260929 01:06:42.830691 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.830060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39022)
0929 01:06:42.830318 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.830431 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 01:06:42.832100 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.831677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39032)
0929 01:06:42.831851 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.831942 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 01:06:42.832540 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:42.839228 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 1
I20260929 01:06:42.840677 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.840037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39036)
0929 01:06:42.840302 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.840444 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:06:42.841873 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.841457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39042)
0929 01:06:42.841631 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.841724 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 01:06:42.842139 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:42.910444 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 2
I20260929 01:06:42.911996 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.911368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39050)
0929 01:06:42.911626 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.911758 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:06:42.913241 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.912821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39054)
0929 01:06:42.913008 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.913106 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 01:06:42.913497 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:42.923837 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 3
I20260929 01:06:42.925356 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.924762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39066)
0929 01:06:42.925026 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.925154 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 01:06:42.926797 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.926287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39076)
0929 01:06:42.926548 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.926644 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260929 01:06:42.927109 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:43.019783 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 3
I20260929 01:06:43.021409 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.020748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39078)
0929 01:06:43.021020 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.021161 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 01:06:43.022934 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.022364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39080)
0929 01:06:43.022646 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.022766 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260929 01:06:43.023231 25658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
I20260929 01:06:43.070379 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 4
I20260929 01:06:43.071926 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.071326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39084)
0929 01:06:43.071606 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.071713 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 01:06:43.073200 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.072801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39088)
0929 01:06:43.072967 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.073059 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 01:06:43.073490 25732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34463: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
W20260929 01:06:43.088197 25740 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:43.088606 25740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:43.088709 25740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:43.088882 25740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:43.096758 25740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:43.096863 25740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:43.096952 25740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:43.097039 25740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:43.097155 25740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:43.107666 25740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:34131,127.3.21.253:39747,127.3.21.252:34463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab
--principal=kudu/127.3.21.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:34463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.21.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:43.109066 25740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:43.110419 25740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:43.124254 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:43.124243 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:43.124570 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:43.125223 25740 server_base.cc:1034] running on GCE node
I20260929 01:06:43.126185 25740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:43.127898 25740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:43.129216 25740 hybrid_clock.cc:648] HybridClock initialized: now 1790644003129140 us; error 70 us; skew 500 ppm
I20260929 01:06:43.129647 25740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:43.133569 25740 init.cc:377] Logged in from keytab as kudu/127.3.21.252@KRBTEST.COM (short username kudu)
I20260929 01:06:43.135679 25740 webserver.cc:460] Webserver started at http://127.3.21.252:32859/ using document root <none> and password file <none>
I20260929 01:06:43.136520 25740 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:43.136683 25740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:43.137377 25740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:43.141242 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data/instance:
uuid: "1de10b110dcc4c4798b12cf7c47d9fcc"
format_stamp: "Formatted at 2026-09-29 01:06:43 on dist-test-slave-k0b7"
server_key: "d270efba739a31eda9183cf2c24b99ce"
server_key_iv: "e8cf4d34097370eabc6fc9c48ffdaa30"
server_key_version: "encryptionkey@0"
I20260929 01:06:43.142237 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal/instance:
uuid: "1de10b110dcc4c4798b12cf7c47d9fcc"
format_stamp: "Formatted at 2026-09-29 01:06:43 on dist-test-slave-k0b7"
server_key: "d270efba739a31eda9183cf2c24b99ce"
server_key_iv: "e8cf4d34097370eabc6fc9c48ffdaa30"
server_key_version: "encryptionkey@0"
I20260929 01:06:43.148911 25740 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:43.152902 25755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:43.154798 25740 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:06:43.155028 25740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal
uuid: "1de10b110dcc4c4798b12cf7c47d9fcc"
format_stamp: "Formatted at 2026-09-29 01:06:43 on dist-test-slave-k0b7"
server_key: "d270efba739a31eda9183cf2c24b99ce"
server_key_iv: "e8cf4d34097370eabc6fc9c48ffdaa30"
server_key_version: "encryptionkey@0"
I20260929 01:06:43.155289 25740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:43.173158 25740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:43.179580 25740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:43.179929 25740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:43.200546 25658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 4
I20260929 01:06:43.201825 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.201248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39102)
0929 01:06:43.201504 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.201643 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 01:06:43.203176 25740 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.252:34463
I20260929 01:06:43.203197 25806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.252:34463 every 8 connection(s)
I20260929 01:06:43.204774 25740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
I20260929 01:06:43.207989 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 25740
I20260929 01:06:43.208201 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643611838815-3159-0/minicluster-data/master-2/wal/instance
I20260929 01:06:43.208793 3159 external_mini_cluster.cc:1428] Setting key f85ac59059b01bc7833216d8e861b3e4
I20260929 01:06:43.209656 25808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:43.220965 25808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:43.227926 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.202819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39108)
0929 01:06:43.202976 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.202997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.203110 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.208289 (+ 5179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.208294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.208714 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.208933 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.208951 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.212329 (+ 3378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.212336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.213745 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.213762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.215530 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.215538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.215721 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.216462 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.216492 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.219356 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.223954 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.223963 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.223968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.224648 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.225251 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.225257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.225261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.225494 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.226754 (+ 1260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.226761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.226945 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.227135 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.227493 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 01:06:43.230959 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.204548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.3.21.252:34463)
0929 01:06:43.205512 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.205543 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.205606 (+ 63us) server_negotiation.cc:407] Connection header received
0929 01:06:43.207449 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.207469 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.207856 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.209162 (+ 1306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.209697 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.212105 (+ 2408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.213931 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.215332 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.215539 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.219717 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.219732 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.219758 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.219806 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.223756 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.224845 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.224855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.224863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.225091 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.225695 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.225703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.225710 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.226596 (+ 886us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.228411 (+ 1815us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.228778 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.229351 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":436,"threads_started":1}
I20260929 01:06:43.231777 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.209830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.3.21.252:34463)
0929 01:06:43.210696 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.210704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.211306 (+ 602us) server_negotiation.cc:407] Connection header received
0929 01:06:43.211527 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.211533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.211605 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.211728 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.213639 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.215350 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.217437 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.218113 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.218260 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.223498 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.223507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.223513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.223560 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.227580 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.229004 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.229010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.229013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.229210 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.229681 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.229690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.229695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.230057 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.230349 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.230978 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.231489 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":665,"thread_start_us":412,"threads_started":1}
I20260929 01:06:43.241680 25658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: Bootstrap starting.
I20260929 01:06:43.246367 25658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:43.248234 25658 log.cc:824] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:06:43.251797 25658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: No bootstrap required, opened a new log
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:06:43.261190 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.224425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:34131 (local address 127.0.0.1:33186)
0929 01:06:43.240034 (+ 15609us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.240105 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.240345 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.242234 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.242249 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.243371 (+ 1122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.243742 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.243758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.244940 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.244946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.246713 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.246739 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.247642 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.247650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.247844 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.248636 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.248661 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.252294 (+ 3633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.257491 (+ 5197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.257510 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.257520 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.258151 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.258714 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.258719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.258723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.258898 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.259644 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.259654 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.259937 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.260222 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.260729 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15230,"thread_start_us":16084,"threads_started":1}
I20260929 01:06:43.261232 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.224396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.3.21.254:34131)
0929 01:06:43.241613 (+ 17217us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.241618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.241636 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:06:43.241690 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.241694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.241758 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.241927 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.243924 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.244747 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.246899 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.247425 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.247575 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.252596 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.252604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.252609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.252647 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.257274 (+ 4627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.258326 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.258332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.258335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.258553 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.259031 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.259037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.259040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.259483 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.259701 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.260379 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.260686 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17096,"thread_start_us":17642,"threads_started":1}
I20260929 01:06:43.262876 25808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:43.263269 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.235627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.3.21.254:34131)
0929 01:06:43.243158 (+ 7531us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.243164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.243189 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:06:43.243240 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.243245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.243318 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.243444 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.245179 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.246588 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.248620 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.249373 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.249513 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.255244 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.255254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.255259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.255319 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.258886 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.259995 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.260002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.260009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.260171 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.260663 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.260670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.260734 (+ 64us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.261091 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.262229 (+ 1138us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.262601 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.262853 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7278,"thread_start_us":7060,"threads_started":1}
I20260929 01:06:43.264760 25658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.265209 25658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363fa8fe58624fcaa8a50985cc8502c1, State: Initialized, Role: FOLLOWER
I20260929 01:06:43.265794 25658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.266836 25817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.267248 25817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.268142 25658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:06:43.281971 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.263670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:39747 (local address 127.0.0.1:42298)
0929 01:06:43.264100 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.264122 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.264264 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.264609 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.264614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.265069 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.265300 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.265315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.267214 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.267221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.268855 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.268873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.270304 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.270311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.270555 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.271770 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.271800 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.274832 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.279072 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.279078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.279082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.279575 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.280087 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.280092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.280095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.280302 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.280895 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.280900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.281086 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.281239 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.281565 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260929 01:06:43.282008 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.264147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.3.21.253:39747)
0929 01:06:43.264349 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.264354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.264370 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:43.264419 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.264422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.264483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.264563 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.265476 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.266784 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.269039 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.270086 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.270260 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.275132 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.275142 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.275150 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.275208 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.278854 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.279739 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.279745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.279751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.279945 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.280439 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.280445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.280448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.280763 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.280933 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.281340 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.281580 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:06:43.283361 25808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:06:43.285174 25612 tablet.cc:2369] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:43.290472 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.265420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.3.21.253:39747)
0929 01:06:43.266079 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.266085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.266111 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:06:43.266218 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.266223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.266298 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.268290 (+ 1992us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.268599 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.269653 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.271904 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.272629 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.273434 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.278964 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.278975 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.278984 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.279041 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.284176 (+ 5135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.287237 (+ 3061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.287244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.287249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.287467 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.288140 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.288146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.288150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.288542 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.288877 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.289472 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.289878 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":312,"threads_started":1}
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:43.298359 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:43.303527 25633 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "363fa8fe58624fcaa8a50985cc8502c1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33188
I20260929 01:06:43.304108 25633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:06:43.304471 25633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:06:43.306567 25828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:43.308164 25828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:43.309345 25633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.311722 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.284068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39126)
0929 01:06:43.284234 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.284251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.284351 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.284745 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.284749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.285151 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.285334 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.285348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.287257 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.287263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.288773 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.288788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.289946 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.289952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.290164 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.291037 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.291062 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.294219 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.308741 (+ 14522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.308748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.308754 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.309337 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.309820 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.309824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.309826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.309981 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.310704 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.310711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.310861 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.310996 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.311341 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:06:43.312805 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.284298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.3.21.252:34463)
0929 01:06:43.284481 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.284486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.284501 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:06:43.284548 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.284551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.284621 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.284730 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.285508 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.286974 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.288947 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.289590 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.289723 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.294452 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.294460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.294466 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.294545 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.308529 (+ 13984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.309534 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.309539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.309543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.309696 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.310107 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.310111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.310114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.310589 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.312144 (+ 1555us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.312281 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.312501 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:06:43.314003 25633 leader_election.cc:288] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [CANDIDATE]: Term 1 election: Requested vote from peers 54395886af59486b81f24c4cec330c83 (127.3.21.253:39747), 1de10b110dcc4c4798b12cf7c47d9fcc (127.3.21.252:34463)
I20260929 01:06:43.298914 25591 master_runner.cc:186] Error getting master registration for 127.3.21.253:39747: OK, instance_id { permanent_uuid: "54395886af59486b81f24c4cec330c83" instance_seqno: 1790644002622835 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:43.315210 25591 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:06:43.318189 25808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: Bootstrap starting.
I20260929 01:06:43.322774 25808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:43.324334 25808 log.cc:824] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:43.329335 25808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: No bootstrap required, opened a new log
I20260929 01:06:43.346890 25808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.347496 25808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de10b110dcc4c4798b12cf7c47d9fcc, State: Initialized, Role: FOLLOWER
I20260929 01:06:43.348085 25808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.370889 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.327185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.3.21.253:39747)
0929 01:06:43.327441 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.327447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.334316 (+ 6869us) server_negotiation.cc:407] Connection header received
0929 01:06:43.335338 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.335344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.335438 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.335574 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.341552 (+ 5978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.342754 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.347355 (+ 4601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.348032 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.348156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.358063 (+ 9907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.358094 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.358101 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.358160 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.362095 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.367244 (+ 5149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.367262 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.367266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.367475 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.368513 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.368519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.368522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.368956 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.369249 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.370213 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.370526 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:06:43.371989 25808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:43.372076 25781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "363fa8fe58624fcaa8a50985cc8502c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc"
I20260929 01:06:43.372695 25781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:43.372720 25732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } attempt: 5
I20260929 01:06:43.374603 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.326983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.3.21.254:34131)
0929 01:06:43.327241 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.327254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.339343 (+ 12089us) server_negotiation.cc:407] Connection header received
0929 01:06:43.339684 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.339691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.339808 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.339895 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.348725 (+ 8830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.350000 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.352639 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.353428 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.353564 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.362868 (+ 9304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.362877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.362882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.362935 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.366998 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.369362 (+ 2364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.369368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.369371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.369543 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.372271 (+ 2728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.372279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.372288 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.372964 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.373245 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.373858 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.374145 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 01:06:43.371907 25838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.375247 25838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.383499 25781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363fa8fe58624fcaa8a50985cc8502c1 in term 1.
I20260929 01:06:43.384321 25610 leader_election.cc:302] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1de10b110dcc4c4798b12cf7c47d9fcc, 363fa8fe58624fcaa8a50985cc8502c1; no voters:
I20260929 01:06:43.384902 25817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 1 FOLLOWER]: Leader election won for term 1
Sep 29 01:06:43 dist-test-slave-k0b7 krb5kdc[25578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:43.404798 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.373335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34463 (local address 127.0.0.1:39150)
0929 01:06:43.374715 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.374738 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.374856 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.375610 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.375615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.376353 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.376806 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.376824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.378988 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.378994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.380946 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.380964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.385426 (+ 4462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.385436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.385686 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.387072 (+ 1386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.387103 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.391194 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.398582 (+ 7388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.398595 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.398603 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.399576 (+ 973us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.400386 (+ 810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.400391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.400395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.400687 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.402164 (+ 1477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.402173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.402452 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.402733 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.403334 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232}
I20260929 01:06:43.405462 25817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [term 1 LEADER]: Becoming Leader. State: Replica: 363fa8fe58624fcaa8a50985cc8502c1, State: Running, Role: LEADER
I20260929 01:06:43.406213 25817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.406742 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.374012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.3.21.252:34463)
0929 01:06:43.375097 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.375104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.375122 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:06:43.375179 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.375185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.375272 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.375369 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.377317 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.378802 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.381129 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.381885 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.382077 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.391678 (+ 9601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.391689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.391697 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.391763 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.398205 (+ 6442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.399849 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.399856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.399860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.400185 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.400942 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.400957 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.400986 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.401878 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.403914 (+ 2036us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.404356 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.406005 (+ 1649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942}
I20260929 01:06:43.411203 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.339581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.3.21.252:34463)
0929 01:06:43.339937 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.339943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.347101 (+ 7158us) server_negotiation.cc:407] Connection header received
0929 01:06:43.347645 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.347652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.347748 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.347836 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.350696 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.351945 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.357428 (+ 5483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.358197 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.358323 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.396396 (+ 38073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.396404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.396413 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.396479 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.400318 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.402298 (+ 1980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.402304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.402307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.402459 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.403369 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.403374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.403377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.403786 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.404017 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.410377 (+ 6360us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.410762 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260929 01:06:43.415395 25840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363fa8fe58624fcaa8a50985cc8502c1. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.416424 25840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:43.420243 25732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: Bootstrap starting.
I20260929 01:06:43.422572 25842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:43.425338 25842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 01:06:43.432444 25761 tablet.cc:2369] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:43.437436 25732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:43.443208 25781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [term 1 FOLLOWER]: Refusing update from remote peer 363fa8fe58624fcaa8a50985cc8502c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:43.444803 25840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:43.440270 25732 log.cc:824] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:43.462138 25732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: No bootstrap required, opened a new log
I20260929 01:06:43.474068 25838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363fa8fe58624fcaa8a50985cc8502c1. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.474504 25838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.481992 25845 mvcc.cc:205] Tried to move back new op lower bound from 7334477838087102464 to 7334477837961428992. Current Snapshot: MvccSnapshot[applied={T|T < 7334477838087102464}]
I20260929 01:06:43.491619 25732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.492169 25732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54395886af59486b81f24c4cec330c83, State: Initialized, Role: FOLLOWER
I20260929 01:06:43.492800 25732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } }
I20260929 01:06:43.496724 25817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.497195 25817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:43.503983 25842 catalog_manager.cc:1347] Generated new cluster ID: 1baa9fcfd62144b49aa9795aa95716e6
I20260929 01:06:43.504148 25842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:43.507124 25858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.507967 25858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.509022 25732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:43.509195 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "363fa8fe58624fcaa8a50985cc8502c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "54395886af59486b81f24c4cec330c83"
I20260929 01:06:43.509824 25707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:43.513696 25707 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363fa8fe58624fcaa8a50985cc8502c1 in term 1.
I20260929 01:06:43.514585 25706 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [term 1 FOLLOWER]: Refusing update from remote peer 363fa8fe58624fcaa8a50985cc8502c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:43.516438 25840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:43.527158 25838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.527480 25838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.535916 25861 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:06:43.553710 25842 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:43.556573 25842 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:43.546645 25740 master_runner.cc:186] Error getting master registration for 127.3.21.253:39747: OK, instance_id { permanent_uuid: "54395886af59486b81f24c4cec330c83" instance_seqno: 1790644002622835 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:43.563023 25740 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:06:43.587808 25842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1: Generated new TSK 0
I20260929 01:06:43.589737 25842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:43.590317 25864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.590794 25864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.591238 25858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363fa8fe58624fcaa8a50985cc8502c1. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.591500 25858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.592769 25840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "363fa8fe58624fcaa8a50985cc8502c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363fa8fe58624fcaa8a50985cc8502c1" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 34131 } } peers { permanent_uuid: "54395886af59486b81f24c4cec330c83" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 39747 } } peers { permanent_uuid: "1de10b110dcc4c4798b12cf7c47d9fcc" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34463 } } }
I20260929 01:06:43.593020 25840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363fa8fe58624fcaa8a50985cc8502c1 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:43.644095 25876 catalog_manager.cc:1259] Loaded cluster ID: 1baa9fcfd62144b49aa9795aa95716e6
I20260929 01:06:43.644222 25876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: loading cluster ID for follower catalog manager: success
I20260929 01:06:43.652737 25876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: acquiring CA information for follower catalog manager: success
I20260929 01:06:43.656438 25876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54395886af59486b81f24c4cec330c83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:06:43.685104 25816 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.677817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.3.21.253:39747)
0929 01:06:43.678020 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.678027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.682395 (+ 4368us) server_negotiation.cc:407] Connection header received
0929 01:06:43.683428 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.683435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.683554 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.683638 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.684807 (+ 1169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42312: BlockingRecv error: recv got EOF from 127.0.0.1:42312 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 01:06:43.685105 25814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.679009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.3.21.254:34131)
0929 01:06:43.679207 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.679213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.682715 (+ 3502us) server_negotiation.cc:407] Connection header received
0929 01:06:43.683033 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.683042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.683154 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.683239 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.684806 (+ 1567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33204: BlockingRecv error: recv got EOF from 127.0.0.1:33204 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 01:06:43.687919 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.683002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.3.21.252:34463)
0929 01:06:43.683209 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.683216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.685847 (+ 2631us) server_negotiation.cc:407] Connection header received
0929 01:06:43.686151 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.686161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.686256 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.686335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.687637 (+ 1302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39158: BlockingRecv error: recv got EOF from 127.0.0.1:39158 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 01:06:43.707544 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.697383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.3.21.253:39747)
0929 01:06:43.697614 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.697621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.699046 (+ 1425us) server_negotiation.cc:407] Connection header received
0929 01:06:43.699329 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.699337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.699619 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.699821 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.700348 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.701698 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.703887 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.704716 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.704873 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.705641 (+ 768us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.706671 (+ 1030us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.706838 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.706993 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.707114 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 01:06:43.723852 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.721147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.3.21.252:34463)
0929 01:06:43.721390 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.721397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.722123 (+ 726us) server_negotiation.cc:407] Connection header received
0929 01:06:43.722409 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.722416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.722601 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.722726 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.723597 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39162: BlockingRecv error: recv got EOF from 127.0.0.1:39162 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 01:06:43.738152 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.730898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.3.21.253:39747)
0929 01:06:43.731105 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.731112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.731230 (+ 118us) server_negotiation.cc:407] Connection header received
0929 01:06:43.731529 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.731536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.731641 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.731752 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.732211 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.733408 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.735562 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.736361 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.736549 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.737183 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.737360 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.737473 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.737675 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.737774 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 01:06:43.756961 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.749461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.3.21.252:34463)
0929 01:06:43.749662 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.749670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.755217 (+ 5547us) server_negotiation.cc:407] Connection header received
0929 01:06:43.755433 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.755440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.755620 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.755768 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.756680 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39164: BlockingRecv error: recv got EOF from 127.0.0.1:39164 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 01:06:43.776762 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.769431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.3.21.253:39747)
0929 01:06:43.769714 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.769721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.769747 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:06:43.769833 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.769839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.769946 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.770086 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.770625 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.771976 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.774229 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.775073 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.775226 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.775784 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.775992 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.776108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.776286 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.776387 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260929 01:06:43.789510 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.785087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.3.21.252:34463)
0929 01:06:43.785292 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.785300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.787545 (+ 2245us) server_negotiation.cc:407] Connection header received
0929 01:06:43.787764 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.787771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.787898 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.788150 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.789157 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39168: BlockingRecv error: recv got EOF from 127.0.0.1:39168 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:06:43.814544 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.804602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.3.21.253:39747)
0929 01:06:43.804872 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.804879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.807153 (+ 2274us) server_negotiation.cc:407] Connection header received
0929 01:06:43.807391 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.807398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.807511 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.807656 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.808262 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.809582 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.811972 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.812683 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.812842 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.813455 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.813664 (+ 209us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.813832 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.814031 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.814138 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
W20260929 01:06:43.825371 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.822649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.3.21.252:34463)
0929 01:06:43.822961 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.822968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.823603 (+ 635us) server_negotiation.cc:407] Connection header received
0929 01:06:43.823873 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.823880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.823987 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.824119 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.825108 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39184: BlockingRecv error: recv got EOF from 127.0.0.1:39184 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 01:06:43.846028 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.837882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.3.21.253:39747)
0929 01:06:43.838105 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.838112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.838705 (+ 593us) server_negotiation.cc:407] Connection header received
0929 01:06:43.838925 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.838933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.839062 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.839179 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.839792 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.840982 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.843469 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.844291 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.844460 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.845000 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.845161 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.845257 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.845455 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.845572 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 01:06:43.864177 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.861620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.3.21.252:34463)
0929 01:06:43.861814 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.861821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.862430 (+ 609us) server_negotiation.cc:407] Connection header received
0929 01:06:43.862663 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.862670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.862782 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.862915 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.863872 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39188: BlockingRecv error: recv got EOF from 127.0.0.1:39188 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:06:43.892768 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.885051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.3.21.253:39747)
0929 01:06:43.885262 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.885269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.885813 (+ 544us) server_negotiation.cc:407] Connection header received
0929 01:06:43.886063 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.886070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.886184 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.886312 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.886885 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.888107 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.890318 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.891090 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.891279 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.891829 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.892004 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.892121 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.892309 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.892417 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 01:06:43.908390 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.905632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.3.21.252:34463)
0929 01:06:43.905847 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.905854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.906657 (+ 803us) server_negotiation.cc:407] Connection header received
0929 01:06:43.906908 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.906915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.907036 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.907182 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.908154 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39198: BlockingRecv error: recv got EOF from 127.0.0.1:39198 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:06:43.958634 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.949923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.3.21.253:39747)
0929 01:06:43.950271 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.950278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.951036 (+ 758us) server_negotiation.cc:407] Connection header received
0929 01:06:43.951315 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.951321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.951438 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.951520 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:43.952533 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.953914 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.956089 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.956845 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.957027 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.957572 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:43.957780 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:43.957876 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.958065 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.958198 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260929 01:06:43.972962 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.970364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.3.21.252:34463)
0929 01:06:43.970612 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.970618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.971244 (+ 626us) server_negotiation.cc:407] Connection header received
0929 01:06:43.971466 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.971472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.971570 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.971681 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.972675 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39214: BlockingRecv error: recv got EOF from 127.0.0.1:39214 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:06:44.048295 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.041100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.3.21.253:39747)
0929 01:06:44.041381 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.041389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.041803 (+ 414us) server_negotiation.cc:407] Connection header received
0929 01:06:44.041978 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.041984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.042090 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.042239 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.042727 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.043996 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.045615 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.046378 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.046594 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.047245 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.047426 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.047535 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.047823 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.047919 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 01:06:44.060600 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:44.058715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.3.21.252:34463)
0929 01:06:44.058906 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.058911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.058930 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:06:44.059057 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.059061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.059138 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.059290 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:44.060325 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39228: BlockingRecv error: recv got EOF from 127.0.0.1:39228 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:06:44.201843 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.196049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.3.21.253:39747)
0929 01:06:44.196310 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.196323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.196522 (+ 199us) server_negotiation.cc:407] Connection header received
0929 01:06:44.196836 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.196842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.196953 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.197039 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.197629 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.198448 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.199978 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.200416 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.200525 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.201021 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.201134 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.201259 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.201444 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.201523 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260929 01:06:44.210520 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:44.208174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.3.21.252:34463)
0929 01:06:44.208423 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.208430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.208803 (+ 373us) server_negotiation.cc:407] Connection header received
0929 01:06:44.208962 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.208967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.209068 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.209171 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:44.210237 (+ 1066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39242: BlockingRecv error: recv got EOF from 127.0.0.1:39242 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:06:44.477399 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.471181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.3.21.253:39747)
0929 01:06:44.471430 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.471434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.471691 (+ 257us) server_negotiation.cc:407] Connection header received
0929 01:06:44.471806 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.471810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.471889 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.471967 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.472379 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.473343 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.475275 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.475879 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.476020 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.476530 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.476664 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.476752 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.477066 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.477139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260929 01:06:44.486613 25810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:44.484788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.3.21.252:34463)
0929 01:06:44.484985 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.484993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.485089 (+ 96us) server_negotiation.cc:407] Connection header received
0929 01:06:44.485331 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.485337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.485440 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.485556 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:44.486342 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39250: BlockingRecv error: recv got EOF from 127.0.0.1:39250 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 01:06:44.540983 25861 catalog_manager.cc:1259] Loaded cluster ID: 1baa9fcfd62144b49aa9795aa95716e6
I20260929 01:06:44.541150 25861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: loading cluster ID for follower catalog manager: success
I20260929 01:06:44.544010 25861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: acquiring CA information for follower catalog manager: success
I20260929 01:06:44.546209 25861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1de10b110dcc4c4798b12cf7c47d9fcc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:06:45.010883 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:45.004728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.3.21.253:39747)
0929 01:06:45.005271 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:45.005276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:45.005297 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:06:45.005370 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:45.005374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:45.005444 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:45.005571 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:45.006164 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.007204 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.008778 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.009352 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.009576 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:45.010008 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:45.010144 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:45.010236 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:45.010414 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 01:06:45.010546 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":333,"threads_started":1}
I20260929 01:06:45.024736 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:45.017531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.3.21.252:34463)
0929 01:06:45.018183 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:45.018188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:45.018358 (+ 170us) server_negotiation.cc:407] Connection header received
0929 01:06:45.018631 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:45.018638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:45.018838 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:45.018955 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:45.019419 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.020216 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.021823 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.022271 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.022384 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:45.023155 (+ 771us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:45.023827 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:45.023968 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:45.024193 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 01:06:45.024275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":254,"threads_started":1}
I20260929 01:06:45.040319 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:45.032057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.3.21.254:34131)
0929 01:06:45.032916 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:45.032925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:45.033595 (+ 670us) server_negotiation.cc:407] Connection header received
0929 01:06:45.033750 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:45.033755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:45.034015 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:45.034224 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:45.034951 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.036000 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.037837 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.038271 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.038423 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:45.039049 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:45.039591 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:45.039708 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:45.039862 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 01:06:45.039959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":563,"threads_started":1}
I20260929 01:06:45.041568 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:45.033701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.3.21.252:34463)
0929 01:06:45.033890 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:45.033895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:45.034550 (+ 655us) server_negotiation.cc:407] Connection header received
0929 01:06:45.034750 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:45.034757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:45.034863 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:45.034943 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:45.035713 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.036815 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.039516 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.040135 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.040260 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:45.040840 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:45.040952 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:45.041064 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:45.041191 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 01:06:45.041273 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:06:45.042577 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:45.032221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.3.21.253:39747)
0929 01:06:45.032466 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:45.032474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:45.033543 (+ 1069us) server_negotiation.cc:407] Connection header received
0929 01:06:45.034407 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:45.034415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:45.034569 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:45.034664 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:45.035667 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.036805 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.039362 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:45.040002 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:45.040140 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:45.041228 (+ 1088us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:45.041358 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:45.041436 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:45.042092 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 01:06:45.042197 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 01:06:45.047043 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25591
I20260929 01:06:45.048193 25653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:45.171178 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25591
I20260929 01:06:45.190696 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25665
I20260929 01:06:45.191926 25727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:45.307790 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25665
I20260929 01:06:45.329394 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 25740
I20260929 01:06:45.330781 25802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:45.447340 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 25740
2026-09-29T01:06:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5395 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:45 dist-test-slave-k0b7 krb5kdc[26012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:45 dist-test-slave-k0b7 krb5kdc[26012](info): set up 2 sockets
Sep 29 01:06:45 dist-test-slave-k0b7 krb5kdc[26012](info): commencing operation
krb5kdc: starting...
W20260929 01:06:46.587679 3159 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/test-user.keytab.
Sep 29 01:06:46 dist-test-slave-k0b7 krb5kdc[26012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.21.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
Entry for principal kudu/127.3.21.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab.
I20260929 01:06:46.899765 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37941
--webserver_interface=127.3.21.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37941,127.3.21.253:42449,127.3.21.252:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:47.255759 26025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:47.256242 26025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:47.256385 26025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:47.256529 26025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:47.264145 26025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:47.264343 26025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:47.264451 26025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:47.264545 26025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:47.264653 26025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:47.275285 26025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37941,127.3.21.253:42449,127.3.21.252:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.254.keytab
--principal=kudu/127.3.21.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.254:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.21.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:47.276733 26025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:47.278095 26025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:47.291433 26030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:47.291375 26033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:47.291486 26031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:47.292378 26025 server_base.cc:1034] running on GCE node
I20260929 01:06:47.293141 26025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:47.294713 26025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:47.295985 26025 hybrid_clock.cc:648] HybridClock initialized: now 1790644007295926 us; error 38 us; skew 500 ppm
I20260929 01:06:47.296365 26025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:47 dist-test-slave-k0b7 krb5kdc[26012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:47.305280 26025 init.cc:377] Logged in from keytab as kudu/127.3.21.254@KRBTEST.COM (short username kudu)
I20260929 01:06:47.307338 26025 webserver.cc:460] Webserver started at http://127.3.21.254:44111/ using document root <none> and password file <none>
I20260929 01:06:47.308255 26025 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:47.308410 26025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:47.308797 26025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:47.312554 26025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data/instance:
uuid: "3eeff2547ff047888432a2b746755e60"
format_stamp: "Formatted at 2026-09-29 01:06:47 on dist-test-slave-k0b7"
server_key: "1e1b513ba513a0b78df572a9f4574711"
server_key_iv: "9ad2781e35d04d09f290752b5f58d5cf"
server_key_version: "encryptionkey@0"
I20260929 01:06:47.313406 26025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal/instance:
uuid: "3eeff2547ff047888432a2b746755e60"
format_stamp: "Formatted at 2026-09-29 01:06:47 on dist-test-slave-k0b7"
server_key: "1e1b513ba513a0b78df572a9f4574711"
server_key_iv: "9ad2781e35d04d09f290752b5f58d5cf"
server_key_version: "encryptionkey@0"
I20260929 01:06:47.319345 26025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:06:47.323173 26040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:47.324539 26025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:06:47.324777 26025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal
uuid: "3eeff2547ff047888432a2b746755e60"
format_stamp: "Formatted at 2026-09-29 01:06:47 on dist-test-slave-k0b7"
server_key: "1e1b513ba513a0b78df572a9f4574711"
server_key_iv: "9ad2781e35d04d09f290752b5f58d5cf"
server_key_version: "encryptionkey@0"
I20260929 01:06:47.325019 26025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:47.345206 26025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:47.350198 26025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:47.350559 26025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:47.372339 26025 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.254:37941
I20260929 01:06:47.372359 26091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.254:37941 every 8 connection(s)
I20260929 01:06:47.373945 26025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:47.378208 26092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:47.379668 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 26025
I20260929 01:06:47.379873 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-0/wal/instance
I20260929 01:06:47.380524 3159 external_mini_cluster.cc:1428] Setting key 34317b118f398a9da7df5883de7d6d3b
I20260929 01:06:47.390136 26092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:47 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.21.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.253@KRBTEST.COM" created.
I20260929 01:06:47.422020 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.394965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.3.21.254:37941)
0929 01:06:47.398787 (+ 3822us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:47.398810 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:47.399843 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 01:06:47.402135 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:47.402142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:47.402428 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:47.402731 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:47.404204 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.405643 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.407384 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.408033 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.408293 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:47.412983 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:47.412993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:47.413007 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:47.413043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:47.416773 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:47.417584 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:47.417590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:47.417592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:47.417777 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:47.418226 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:47.418237 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:47.418240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:47.418861 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:47.419083 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:47.419661 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 01:06:47.420118 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":3521,"thread_start_us":3224,"threads_started":1}
I20260929 01:06:47.422056 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.394947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37941 (local address 127.0.0.1:34450)
0929 01:06:47.399280 (+ 4333us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.399346 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:47.401596 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:47.402761 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:47.402769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:47.403571 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:47.403950 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:47.403969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.405830 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.405837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:47.407225 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:47.407241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.408317 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.408325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:47.408514 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:47.409501 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:47.409529 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:47.412487 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:47.416932 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:47.416946 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:47.416960 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:47.417420 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:47.417893 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:47.417900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:47.417906 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:47.418108 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:47.418998 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:47.419011 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:47.419257 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 01:06:47.419512 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 01:06:47.420250 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3978,"mutex_wait_us":228,"thread_start_us":3726,"threads_started":1}
I20260929 01:06:47.425814 26092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:47.427274 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.426812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42888)
0929 01:06:47.427029 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.427142 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 01:06:47.427554 26041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
I20260929 01:06:47.429687 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.429261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42896)
0929 01:06:47.429421 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.429516 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:06:47.429950 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:42449: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab.
Entry for principal kudu/127.3.21.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab.
I20260929 01:06:47.455590 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:42449
--webserver_interface=127.3.21.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab
--principal=kudu/127.3.21.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37941,127.3.21.253:42449,127.3.21.252:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:06:47.479365 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } attempt: 1
I20260929 01:06:47.480643 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.480191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42908)
0929 01:06:47.480396 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.480513 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 01:06:47.481652 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.481278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42912)
0929 01:06:47.481436 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.481505 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:06:47.481835 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:42449: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
I20260929 01:06:47.550148 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } attempt: 2
I20260929 01:06:47.551427 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.550965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42920)
0929 01:06:47.551181 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.551283 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 01:06:47.552539 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.552101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42930)
0929 01:06:47.552308 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.552407 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:06:47.552781 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:42449: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
I20260929 01:06:47.644099 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } attempt: 3
I20260929 01:06:47.645331 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.644848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42942)
0929 01:06:47.645056 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.645158 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 01:06:47.646351 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.646000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42954)
0929 01:06:47.646144 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.646221 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:06:47.646586 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:42449: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
I20260929 01:06:47.789891 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } attempt: 4
I20260929 01:06:47.791273 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.790743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42968)
0929 01:06:47.791030 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.791138 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 01:06:47.792317 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.791965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42972)
0929 01:06:47.792113 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.792199 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 01:06:47.792536 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.253:42449: Network error: Client connection negotiation failed: client connection to 127.3.21.253:42449: connect: Connection refused (error 111)
W20260929 01:06:47.803205 26099 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:47.803606 26099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:47.803725 26099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:47.803833 26099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:47.811515 26099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:47.811630 26099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:47.811715 26099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:47.811802 26099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:47.811904 26099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:47.822675 26099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37941,127.3.21.253:42449,127.3.21.252:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.253.keytab
--principal=kudu/127.3.21.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.253:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.21.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:47.824085 26099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:47.825428 26099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:47.839814 26104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:47.839730 26105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:47.839655 26107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:47.840553 26099 server_base.cc:1034] running on GCE node
I20260929 01:06:47.841423 26099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:47.843056 26099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:47.844358 26099 hybrid_clock.cc:648] HybridClock initialized: now 1790644007844262 us; error 63 us; skew 500 ppm
I20260929 01:06:47.844933 26099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:47 dist-test-slave-k0b7 krb5kdc[26012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:47.848758 26099 init.cc:377] Logged in from keytab as kudu/127.3.21.253@KRBTEST.COM (short username kudu)
I20260929 01:06:47.850472 26099 webserver.cc:460] Webserver started at http://127.3.21.253:33515/ using document root <none> and password file <none>
I20260929 01:06:47.851289 26099 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:47.851382 26099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:47.851665 26099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:47.855139 26099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data/instance:
uuid: "e06e47a3a71940c6b5c5ec2f705b0f12"
format_stamp: "Formatted at 2026-09-29 01:06:47 on dist-test-slave-k0b7"
server_key: "2949cd892f240d3666b3b107e0f0d5b5"
server_key_iv: "20efdb5953d0c23be78486b44fe0728d"
server_key_version: "encryptionkey@0"
I20260929 01:06:47.856005 26099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal/instance:
uuid: "e06e47a3a71940c6b5c5ec2f705b0f12"
format_stamp: "Formatted at 2026-09-29 01:06:47 on dist-test-slave-k0b7"
server_key: "2949cd892f240d3666b3b107e0f0d5b5"
server_key_iv: "20efdb5953d0c23be78486b44fe0728d"
server_key_version: "encryptionkey@0"
I20260929 01:06:47.861904 26099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:47.865962 26114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:47.867686 26099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:47.867864 26099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal
uuid: "e06e47a3a71940c6b5c5ec2f705b0f12"
format_stamp: "Formatted at 2026-09-29 01:06:47 on dist-test-slave-k0b7"
server_key: "2949cd892f240d3666b3b107e0f0d5b5"
server_key_iv: "20efdb5953d0c23be78486b44fe0728d"
server_key_version: "encryptionkey@0"
I20260929 01:06:47.868075 26099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:47.906982 26099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:47.911783 26099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:47.912029 26099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:47.933350 26099 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.253:42449
I20260929 01:06:47.933401 26165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.253:42449 every 8 connection(s)
I20260929 01:06:47.934842 26099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/data/info.pb
I20260929 01:06:47.935184 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 26099
I20260929 01:06:47.935369 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-1/wal/instance
I20260929 01:06:47.935953 3159 external_mini_cluster.cc:1428] Setting key 0363e7a3050e271c4c999b2dcadaff9f
I20260929 01:06:47.939126 26166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:47.945873 26166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.21.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.21.252@KRBTEST.COM" created.
Sep 29 01:06:47 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
I20260929 01:06:47.971542 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.949091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.3.21.254:37941)
0929 01:06:47.949935 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:47.949944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:47.950624 (+ 680us) server_negotiation.cc:407] Connection header received
0929 01:06:47.952792 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:47.952798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:47.952905 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:47.953042 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:47.954973 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.955730 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.957448 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.957908 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.958063 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:47.964486 (+ 6423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:47.964491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:47.964495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:47.964589 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:47.967859 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:47.968837 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:47.968842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:47.968845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:47.968982 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:47.969524 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:47.969528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:47.969531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:47.969848 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:47.970025 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:47.971007 (+ 982us) server_negotiation.cc:299] Negotiation successful
0929 01:06:47.971257 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":541,"threads_started":1}
I20260929 01:06:47.972980 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.949480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37941 (local address 127.0.0.1:34460)
0929 01:06:47.950382 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.950424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:47.952458 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:47.953324 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:47.953333 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:47.953928 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:47.954681 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:47.954706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.955931 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.955935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:47.957264 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:47.957279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.958186 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.958194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:47.958446 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:47.959286 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:47.959313 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:47.964191 (+ 4878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:47.968081 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:47.968091 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:47.968108 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:47.968647 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:47.969117 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:47.969125 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:47.969134 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:47.969360 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:47.970009 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:47.970018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:47.970420 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 01:06:47.970868 (+ 448us) client_negotiation.cc:240] Negotiation successful
0929 01:06:47.971579 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":393,"threads_started":1}
I20260929 01:06:47.975011 26166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:47 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:06:47.991320 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.975674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42980)
0929 01:06:47.975877 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.975907 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:47.975996 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:47.977183 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:47.977187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:47.977809 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:47.977957 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:47.977968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.979494 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.979500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:47.980402 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:47.980415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.981322 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.981329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:47.981558 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:47.982240 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:47.982260 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:47.984521 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:47.988216 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:47.988222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:47.988227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:47.988646 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:47.989081 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:47.989087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:47.989091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:47.989270 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:47.990159 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:47.990163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:47.990284 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 01:06:47.990419 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 01:06:47.990944 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 01:06:47.991432 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.975982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.3.21.253:42449)
0929 01:06:47.976686 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:47.976693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:47.976721 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:06:47.976771 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:47.976778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:47.976961 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:47.977193 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:47.978158 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.979330 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.980560 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:47.981159 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:47.981311 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:47.984773 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:47.984788 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:47.984803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:47.984840 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:47.988034 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:47.988803 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:47.988818 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:47.988820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:47.988949 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:47.989390 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:47.989398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:47.989401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:47.989951 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:47.990196 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:47.990815 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 01:06:47.991046 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":548,"thread_start_us":388,"threads_started":1}
I20260929 01:06:47.993891 26166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:47.995421 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.994751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40822)
0929 01:06:47.995086 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.995253 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260929 01:06:47.995831 26116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:47.997781 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:47.997217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40836)
0929 01:06:47.997403 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:47.997503 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.21.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab.
Entry for principal kudu/127.3.21.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab.
W20260929 01:06:47.998080 26166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:47.999089 3159 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
/tmp/dist-test-taskmEC8aK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:34003
--webserver_interface=127.3.21.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab
--principal=kudu/127.3.21.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.21.212:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.21.254:37941,127.3.21.253:42449,127.3.21.252:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:06:48.047461 26166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 1
I20260929 01:06:48.048842 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.048340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40848)
0929 01:06:48.048543 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.048665 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 01:06:48.049948 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.049586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40856)
0929 01:06:48.049759 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.049835 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:06:48.050176 26166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.091811 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } attempt: 5
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:06:48.105979 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.092641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42982)
0929 01:06:48.092944 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.092967 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:48.093155 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:48.093544 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:48.093549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:48.093982 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:48.094172 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.094181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.095255 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.095260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.096179 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.096190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.096966 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.096971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.097182 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.097847 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:48.097867 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:48.100043 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:48.103378 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.103384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.103389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.103816 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.104233 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.104238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.104242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.104396 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.104949 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:48.104954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:48.105122 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 01:06:48.105299 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 01:06:48.105617 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260929 01:06:48.105981 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.092752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.3.21.253:42449)
0929 01:06:48.092963 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.092970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.093063 (+ 93us) server_negotiation.cc:407] Connection header received
0929 01:06:48.093318 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.093323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.093392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.093503 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.094330 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.095109 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.096330 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.096806 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.096953 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.100284 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.100289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.100293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.100330 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.103208 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.103954 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.103960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.103966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.104100 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.104513 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.104519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.104522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.104824 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.104986 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.105411 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.105648 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:06:48.107491 26092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:48.108966 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.108425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40870)
0929 01:06:48.108609 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.108813 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 01:06:48.110045 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.109707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40886)
0929 01:06:48.109853 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.109940 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:06:48.110280 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.118427 26166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 2
I20260929 01:06:48.119938 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.119335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40896)
0929 01:06:48.119565 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.119681 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 01:06:48.121320 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.120810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40900)
0929 01:06:48.121091 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.121181 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260929 01:06:48.121635 26166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.161628 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 1
I20260929 01:06:48.163136 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.162606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40914)
0929 01:06:48.162844 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.162946 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 01:06:48.164561 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.164160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40920)
0929 01:06:48.164347 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.164429 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 01:06:48.164943 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.212955 26166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 3
I20260929 01:06:48.214416 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.213863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40922)
0929 01:06:48.214145 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.214240 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 01:06:48.215803 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.215241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40938)
0929 01:06:48.215556 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.215678 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260929 01:06:48.216132 26166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.233214 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 2
I20260929 01:06:48.234905 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.234209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40948)
0929 01:06:48.234461 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.234724 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 01:06:48.236054 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.235672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40960)
0929 01:06:48.235830 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.235917 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:06:48.236303 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.342633 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 3
I20260929 01:06:48.344130 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.343437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40968)
0929 01:06:48.343710 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.343827 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 01:06:48.345424 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.344947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40976)
0929 01:06:48.345188 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.345285 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 01:06:48.345824 26092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
I20260929 01:06:48.359378 26166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 4
I20260929 01:06:48.360616 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.360101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40984)
0929 01:06:48.360351 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.360437 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 01:06:48.362316 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.361737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:40988)
0929 01:06:48.361985 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.362123 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 01:06:48.362632 26166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.21.252:34003: Network error: Client connection negotiation failed: client connection to 127.3.21.252:34003: connect: Connection refused (error 111)
W20260929 01:06:48.369813 26174 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:48.370165 26174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:48.370314 26174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:48.370415 26174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:48.378448 26174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:48.378650 26174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:48.378775 26174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:48.378859 26174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:48.378962 26174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:48.389465 26174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.21.212:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.21.254:37941,127.3.21.253:42449,127.3.21.252:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/krb5kdc/kudu_127.3.21.252.keytab
--principal=kudu/127.3.21.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.21.252:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.21.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 23:53:35 UTC on ccycloud.quasar-vwuirl.root.comops.site
build id 4678654
ASAN enabled
I20260929 01:06:48.391008 26174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:48.392378 26174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:48.406895 26179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:48.406847 26180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:48.406950 26182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:48.407660 26174 server_base.cc:1034] running on GCE node
I20260929 01:06:48.408540 26174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:48.410148 26174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:48.411432 26174 hybrid_clock.cc:648] HybridClock initialized: now 1790644008411289 us; error 81 us; skew 500 ppm
I20260929 01:06:48.411890 26174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:48.415874 26174 init.cc:377] Logged in from keytab as kudu/127.3.21.252@KRBTEST.COM (short username kudu)
I20260929 01:06:48.417620 26174 webserver.cc:460] Webserver started at http://127.3.21.252:35623/ using document root <none> and password file <none>
I20260929 01:06:48.418393 26174 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:48.418545 26174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:48.418926 26174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:48.422289 26174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data/instance:
uuid: "786663846d6645f89c3353683fe7acac"
format_stamp: "Formatted at 2026-09-29 01:06:48 on dist-test-slave-k0b7"
server_key: "6d9f4eb35261e77848236aa224be53a8"
server_key_iv: "7b20e00033fa824420057f8a489d6cf9"
server_key_version: "encryptionkey@0"
I20260929 01:06:48.423285 26174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal/instance:
uuid: "786663846d6645f89c3353683fe7acac"
format_stamp: "Formatted at 2026-09-29 01:06:48 on dist-test-slave-k0b7"
server_key: "6d9f4eb35261e77848236aa224be53a8"
server_key_iv: "7b20e00033fa824420057f8a489d6cf9"
server_key_version: "encryptionkey@0"
I20260929 01:06:48.429589 26174 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260929 01:06:48.433614 26189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:48.435247 26174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:48.435464 26174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data,/tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal
uuid: "786663846d6645f89c3353683fe7acac"
format_stamp: "Formatted at 2026-09-29 01:06:48 on dist-test-slave-k0b7"
server_key: "6d9f4eb35261e77848236aa224be53a8"
server_key_iv: "7b20e00033fa824420057f8a489d6cf9"
server_key_version: "encryptionkey@0"
I20260929 01:06:48.435693 26174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:48.453397 26174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:48.458249 26174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:48.458612 26174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:48.479240 26174 rpc_server.cc:307] RPC server started. Bound to: 127.3.21.252:34003
I20260929 01:06:48.479272 26240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.21.252:34003 every 8 connection(s)
I20260929 01:06:48.480674 26174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/data/info.pb
I20260929 01:06:48.484505 26241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:48.489156 3159 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu as pid 26174
I20260929 01:06:48.489403 3159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmEC8aK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643611838815-3159-0/minicluster-data/master-2/wal/instance
I20260929 01:06:48.489856 3159 external_mini_cluster.cc:1428] Setting key 47b56499784bcd52620940880e947982
I20260929 01:06:48.492106 26241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:48.514961 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.494870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.3.21.254:37941)
0929 01:06:48.495492 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.495500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.496180 (+ 680us) server_negotiation.cc:407] Connection header received
0929 01:06:48.496255 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.496260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.496345 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.496470 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.498233 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.499486 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.501892 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.502519 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.502641 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.506814 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.506823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.506828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.506872 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.510937 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.511964 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.511971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.511975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.512200 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.512701 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.512708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.512712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.513012 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.513128 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.514208 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.514480 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
I20260929 01:06:48.515939 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.494873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.254:37941 (local address 127.0.0.1:34468)
0929 01:06:48.495754 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.495789 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:48.496042 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:48.496582 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:48.496589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:48.497487 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:48.498046 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.498063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.499674 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.499678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.501735 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.501747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.502780 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.502788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.502957 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.503679 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:48.503710 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:48.506602 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:48.511146 (+ 4544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.511157 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.511168 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.511781 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.512358 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.512361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.512364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.512562 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.513167 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:48.513178 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:48.513627 (+ 449us) client_negotiation.cc:769] Sending connection context
0929 01:06:48.513970 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 01:06:48.514459 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":40,"spinlock_wait_cycles":19200,"thread_start_us":531,"threads_started":1}
I20260929 01:06:48.516134 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.491837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.3.21.252:34003)
0929 01:06:48.492742 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.492755 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.492796 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:06:48.494876 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.494902 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.495526 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.495924 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.497811 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.499634 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.501581 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.502384 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.502617 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.507909 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.507922 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.507941 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.507986 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.512566 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.513455 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.513462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.513464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.513706 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.514164 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.514170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.514173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.514809 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.515249 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.515441 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.515705 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":672,"thread_start_us":437,"threads_started":1}
I20260929 01:06:48.518008 26241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:48.523044 26092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 4
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.254@KRBTEST.COM
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.254@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:48.536365 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.518767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.253:42449 (local address 127.0.0.1:42992)
0929 01:06:48.518958 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.518979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:48.519113 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:48.519629 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:48.519633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:48.520007 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:48.520164 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.520172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.521594 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.521600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.523018 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.523037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.524150 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.524155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.524374 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.525060 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:48.525086 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:48.527854 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:48.532686 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.532697 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.532704 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.533278 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.533808 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.533812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.533814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.534093 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.535172 (+ 1079us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:48.535177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:48.535360 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 01:06:48.535549 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 01:06:48.535927 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":51584}
I20260929 01:06:48.536612 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.518996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.3.21.253:42449)
0929 01:06:48.519199 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.519207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.519224 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:06:48.519286 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.519291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.519368 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.519462 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.520324 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.521436 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.523183 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.523999 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.524146 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.528137 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.528145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.528154 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.528206 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.532494 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.533424 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.533434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.533439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.533637 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.534233 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.534242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.534247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.535044 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.535222 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.535662 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.536176 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:06:48.537738 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.519355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.3.21.254:37941)
0929 01:06:48.519585 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.519591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.519610 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:06:48.519727 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.519732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.519812 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.519915 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.521437 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.522605 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.524487 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.525172 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.525304 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.530209 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.530218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.530224 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.530259 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.533687 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.534585 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.534592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.534597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.534765 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.535293 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.535300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.535304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.535838 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.536124 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.536444 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.536699 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":107}
I20260929 01:06:48.537843 26241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:48.542299 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.523877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.3.21.252:34003)
0929 01:06:48.524035 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.524040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.524054 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:06:48.524102 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.524107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.524184 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.524336 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.525448 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.526559 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.528309 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.528986 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.529130 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.533778 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.533785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.533792 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.533833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.538981 (+ 5148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.539907 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.539913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.539917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.540090 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.540541 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.540547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.540550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.540981 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.541130 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.541628 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.541914 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:48.542309 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.523625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:41000)
0929 01:06:48.523803 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.523822 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:48.523917 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:48.524397 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:48.524405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:48.525010 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:48.525255 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.525273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.526718 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.526723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.528157 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.528169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.529191 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.529199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.529439 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.530022 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:48.530046 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:48.533567 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:48.539179 (+ 5612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.539187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.539193 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.539803 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.540207 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.540212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.540215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.540414 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.541108 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:48.541113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:48.541316 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 01:06:48.541498 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 01:06:48.541878 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":23}
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.252@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.21.253@KRBTEST.COM
I20260929 01:06:48.549795 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: Bootstrap starting.
I20260929 01:06:48.554072 26092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:48.555351 26092 log.cc:824] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:48.557789 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.538774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:41004)
0929 01:06:48.539151 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.539171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:48.539280 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:48.540633 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:48.540637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:48.541190 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:48.541398 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.541409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.543175 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.543180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.544572 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.544585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.545790 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.545796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.545991 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.546993 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:48.547018 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:48.550185 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:48.554515 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.554521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.554526 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.555132 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.555550 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.555557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.555563 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.555722 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.556438 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:48.556446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:48.556647 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 01:06:48.556928 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 01:06:48.557302 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260929 01:06:48.557927 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.538943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.3.21.252:34003)
0929 01:06:48.539989 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.539997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.540022 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:06:48.540080 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.540086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.540163 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.540357 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.541576 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.543015 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.544729 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.545450 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.545583 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.550432 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.550438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.550441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.550474 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.554317 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.555278 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.555283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.555287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.555428 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.555838 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.555844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.555850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.556270 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.556451 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.557072 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.557448 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":913,"thread_start_us":741,"threads_started":1}
I20260929 01:06:48.559185 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: No bootstrap required, opened a new log
I20260929 01:06:48.559654 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.540232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.3.21.253:42449)
0929 01:06:48.540451 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.540459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.540861 (+ 402us) server_negotiation.cc:407] Connection header received
0929 01:06:48.541013 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.541021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.541102 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.541219 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.543012 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.544144 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.545566 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.546024 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.546175 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.551659 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.551667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.551671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.551718 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.555774 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.556699 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.556705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.556708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.556923 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.557414 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.557419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.557422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.557826 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.558091 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.558871 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.559313 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 01:06:48.564085 26241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: Bootstrap starting.
I20260929 01:06:48.565443 3159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:48.568881 26241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:48.569275 26067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3eeff2547ff047888432a2b746755e60"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478
I20260929 01:06:48.569691 26092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.570166 26092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eeff2547ff047888432a2b746755e60, State: Initialized, Role: FOLLOWER
I20260929 01:06:48.570472 26241 log.cc:824] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:48.570777 26092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.572314 26249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.572897 26249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.573490 26092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:48.573638 26067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:06:48.573875 26067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:48.574723 26241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: No bootstrap required, opened a new log
I20260929 01:06:48.577818 26067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.582101 26067 leader_election.cc:288] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [CANDIDATE]: Term 1 election: Requested vote from peers e06e47a3a71940c6b5c5ec2f705b0f12 (127.3.21.253:42449), 786663846d6645f89c3353683fe7acac (127.3.21.252:34003)
I20260929 01:06:48.587838 26241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.588418 26241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786663846d6645f89c3353683fe7acac, State: Initialized, Role: FOLLOWER
I20260929 01:06:48.589084 26241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.593613 26266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.594168 26266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.596607 26241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:48.596655 26216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eeff2547ff047888432a2b746755e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "786663846d6645f89c3353683fe7acac"
I20260929 01:06:48.597223 26216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:48.602335 26216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eeff2547ff047888432a2b746755e60 in term 1.
I20260929 01:06:48.603632 26042 leader_election.cc:302] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eeff2547ff047888432a2b746755e60, 786663846d6645f89c3353683fe7acac; no voters:
W20260929 01:06:48.604024 26046 tablet.cc:2369] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:48.604451 26249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:48.605274 26249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [term 1 LEADER]: Becoming Leader. State: Replica: 3eeff2547ff047888432a2b746755e60, State: Running, Role: LEADER
I20260929 01:06:48.606006 26249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.614259 26271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eeff2547ff047888432a2b746755e60. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.614866 26271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:48.615654 26025 master_runner.cc:186] Error getting master registration for 127.3.21.253:42449: OK, instance_id { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" instance_seqno: 1790644007927258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:48.631733 26025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:06:48.636255 26269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:48.636411 26269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:48.636550 26274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:48.638949 26274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:48.660490 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.589120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.3.21.252:34003)
0929 01:06:48.589384 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.589392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.617364 (+ 27972us) server_negotiation.cc:407] Connection header received
0929 01:06:48.617926 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.617935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.618065 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.618162 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.629607 (+ 11445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.631517 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.640554 (+ 9037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.641303 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.641450 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.647730 (+ 6280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.647738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.647742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.647795 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.651903 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.656214 (+ 4311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.656225 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.656232 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.656441 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.658606 (+ 2165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.658612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.658615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.658984 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.659218 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.659903 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.660165 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:06:48.661271 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.589572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.3.21.254:37941)
0929 01:06:48.589817 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.589827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.620703 (+ 30876us) server_negotiation.cc:407] Connection header received
0929 01:06:48.621530 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.621538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.621656 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.621762 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.635399 (+ 13637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.636825 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.641397 (+ 4572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.642143 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.642259 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.649964 (+ 7705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.649971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.649976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.650023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.654389 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.656313 (+ 1924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.656321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.656324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.656640 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.657446 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.657483 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.657492 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.658329 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.658819 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.660001 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.660446 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 01:06:48.661888 26166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } attempt: 5
I20260929 01:06:48.671695 26216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [term 1 FOLLOWER]: Refusing update from remote peer 3eeff2547ff047888432a2b746755e60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:48.673125 26271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 01:06:48 dist-test-slave-k0b7 krb5kdc[26012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644007, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.21.253@KRBTEST.COM for kudu/127.3.21.252@KRBTEST.COM
I20260929 01:06:48.692492 26266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eeff2547ff047888432a2b746755e60. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.692842 26266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.693732 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.619955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.3.21.253:42449)
0929 01:06:48.620217 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.620223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.633365 (+ 13142us) server_negotiation.cc:407] Connection header received
0929 01:06:48.636834 (+ 3469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.636841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.636934 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.637010 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.643811 (+ 6801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.645016 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.649883 (+ 4867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.650682 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.650793 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.662905 (+ 12112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.662915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.662919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.662960 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.668629 (+ 5669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.688424 (+ 19795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.688435 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.688442 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.688711 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.689813 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.689822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.689828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.690408 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.690925 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.692861 (+ 1936us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.693207 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:06:48.697433 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.662503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.21.252:34003 (local address 127.0.0.1:41026)
0929 01:06:48.662861 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:48.662882 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:48.662967 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:48.669448 (+ 6481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:48.669457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:48.670152 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:48.670450 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.670467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.674434 (+ 3967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.674444 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.676205 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:48.676232 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.677932 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.677944 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:48.678280 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.679214 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:48.679251 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:48.682533 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:48.691498 (+ 8965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.691504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.691508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.692151 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.694852 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:48.694860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:48.694865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:48.695067 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:48.695999 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:48.696008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:48.696246 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 01:06:48.696430 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 01:06:48.696867 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260929 01:06:48.699970 26280 mvcc.cc:205] Tried to move back new op lower bound from 7334477859472449536 to 7334477859257692160. Current Snapshot: MvccSnapshot[applied={T|T < 7334477859472449536}]
I20260929 01:06:48.705865 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:48.663175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.3.21.252:34003)
0929 01:06:48.668988 (+ 5813us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.668997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.669022 (+ 25us) server_negotiation.cc:407] Connection header received
0929 01:06:48.669092 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.669101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.669210 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.670680 (+ 1470us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.672822 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.673921 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.676485 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:48.677384 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:48.677550 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:48.682847 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:48.682853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:48.682858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:48.682898 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:48.689611 (+ 6713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.692360 (+ 2749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.692365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.692368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.692534 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:48.695250 (+ 2716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:48.695255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:48.695258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:48.695678 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:48.698004 (+ 2326us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:48.698200 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 01:06:48.698584 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5630}
I20260929 01:06:48.710837 26266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.711272 26266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.713348 26166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: Bootstrap starting.
I20260929 01:06:48.717896 26274 catalog_manager.cc:1347] Generated new cluster ID: d5f30e44defa498390ecc21b7f65b971
I20260929 01:06:48.718019 26274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:48.719053 26249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.719435 26249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:48.721153 26166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:48.722560 26166 log.cc:824] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:48.734545 26166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: No bootstrap required, opened a new log
I20260929 01:06:48.740918 26274 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:48.742835 26274 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:48.750356 26166 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.757015 26166 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06e47a3a71940c6b5c5ec2f705b0f12, State: Initialized, Role: FOLLOWER
I20260929 01:06:48.758944 26166 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } }
I20260929 01:06:48.774183 26290 catalog_manager.cc:1259] Loaded cluster ID: d5f30e44defa498390ecc21b7f65b971
I20260929 01:06:48.774317 26290 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: loading cluster ID for follower catalog manager: success
I20260929 01:06:48.774524 26274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60: Generated new TSK 0
I20260929 01:06:48.775703 26274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:48.778224 26290 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: acquiring CA information for follower catalog manager: success
I20260929 01:06:48.779927 26292 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.780543 26292 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.781037 26290 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 786663846d6645f89c3353683fe7acac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:06:48.783015 26141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eeff2547ff047888432a2b746755e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12"
I20260929 01:06:48.783521 26140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:48.782990 26166 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:48.761474 26174 master_runner.cc:186] Error getting master registration for 127.3.21.253:42449: OK, instance_id { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" instance_seqno: 1790644007927258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:48.788028 26174 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:06:48.795866 26140 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [term 1 FOLLOWER]: Refusing update from remote peer 3eeff2547ff047888432a2b746755e60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:48.797927 26249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:48.799675 26141 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eeff2547ff047888432a2b746755e60 in term 1.
I20260929 01:06:48.871140 26292 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eeff2547ff047888432a2b746755e60. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.871564 26292 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.879103 26300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:48.879753 26300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:48.898000 26309 catalog_manager.cc:1259] Loaded cluster ID: d5f30e44defa498390ecc21b7f65b971
I20260929 01:06:48.898135 26309 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: loading cluster ID for follower catalog manager: success
I20260929 01:06:48.902747 26309 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: acquiring CA information for follower catalog manager: success
I20260929 01:06:48.906196 26309 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e06e47a3a71940c6b5c5ec2f705b0f12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:06:48.910022 3159 mini_kdc.cc:348] Time spent kdestroy: real 0.113s user 0.001s sys 0.025s
W20260929 01:06:48.967371 26245 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:48.964255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.3.21.254:37941)
0929 01:06:48.964495 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.964503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.965339 (+ 836us) server_negotiation.cc:407] Connection header received
0929 01:06:48.965628 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.965634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.965732 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.965811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.967100 (+ 1289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34490: BlockingRecv error: recv got EOF from 127.0.0.1:34490 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 01:06:48.969858 26247 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:48.965154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.3.21.252:34003)
0929 01:06:48.965319 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.965324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.968038 (+ 2714us) server_negotiation.cc:407] Connection header received
0929 01:06:48.968339 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.968345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.968435 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.968510 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.969580 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41032: BlockingRecv error: recv got EOF from 127.0.0.1:41032 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 01:06:48.983815 26172 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:48.980563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.3.21.253:42449)
0929 01:06:48.980814 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:48.980821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:48.981510 (+ 689us) server_negotiation.cc:407] Connection header received
0929 01:06:48.981821 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:48.981828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:48.981925 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:48.982007 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:48.983534 (+ 1527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43010: BlockingRecv error: recv got EOF from 127.0.0.1:43010 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:06:49.031574 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:49.012490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.3.21.253:42449)
0929 01:06:49.012697 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:49.012704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:49.023191 (+ 10487us) server_negotiation.cc:407] Connection header received
0929 01:06:49.023514 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:49.023524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:49.023761 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:49.023921 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:49.024713 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.025920 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.028127 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.028841 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.028975 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:49.029903 (+ 928us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:49.030692 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:49.030842 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:49.031094 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 01:06:49.031204 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:06:49.086366 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:49.066943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.3.21.252:34003)
0929 01:06:49.067228 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:49.067237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:49.077828 (+ 10591us) server_negotiation.cc:407] Connection header received
0929 01:06:49.078474 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:49.078503 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:49.078751 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:49.078919 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:49.079675 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.080923 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.083084 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.083833 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.083942 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:49.084748 (+ 806us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:49.085465 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:49.085591 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:49.085832 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 01:06:49.085934 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 01:06:49.160058 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:49.121044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.3.21.254:37941)
0929 01:06:49.121273 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:49.121282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:49.144119 (+ 22837us) server_negotiation.cc:407] Connection header received
0929 01:06:49.144745 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:49.144752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:49.145000 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:49.145171 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:49.146017 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.147404 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.152711 (+ 5307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.153548 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.153664 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:49.158393 (+ 4729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:49.158979 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:49.159104 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:49.159566 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 01:06:49.159687 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 01:06:49.169781 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 26025
I20260929 01:06:49.171060 26087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:49.179694 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:49.130211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.3.21.252:34003)
0929 01:06:49.130412 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:49.130419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:49.150021 (+ 19602us) server_negotiation.cc:407] Connection header received
0929 01:06:49.150348 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:49.150354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:49.150451 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:49.150558 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:49.154400 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.155712 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.174035 (+ 18323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.175197 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.175362 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:49.178425 (+ 3063us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:49.178689 (+ 264us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:49.178779 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:49.179078 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 01:06:49.179223 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:06:49.186405 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:49.152034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.3.21.253:42449)
0929 01:06:49.152267 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:49.152273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:49.161129 (+ 8856us) server_negotiation.cc:407] Connection header received
0929 01:06:49.162125 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:49.162132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:49.162277 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:49.162348 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:49.171052 (+ 8704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.172722 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.183186 (+ 10464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:49.184204 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:49.184342 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:49.185197 (+ 855us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:49.185458 (+ 261us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:49.185554 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:49.185823 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 01:06:49.185953 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 01:06:49.320307 26271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eeff2547ff047888432a2b746755e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eeff2547ff047888432a2b746755e60" member_type: VOTER last_known_addr { host: "127.3.21.254" port: 37941 } } peers { permanent_uuid: "e06e47a3a71940c6b5c5ec2f705b0f12" member_type: VOTER last_known_addr { host: "127.3.21.253" port: 42449 } } peers { permanent_uuid: "786663846d6645f89c3353683fe7acac" member_type: VOTER last_known_addr { host: "127.3.21.252" port: 34003 } } }
I20260929 01:06:49.320621 26271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eeff2547ff047888432a2b746755e60 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:49.343905 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 26025
I20260929 01:06:49.365514 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 26099
I20260929 01:06:49.366732 26161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:49.487905 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 26099
I20260929 01:06:49.508491 3159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu pid 26174
I20260929 01:06:49.509763 26236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:06:49.622323 3159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmEC8aK/build/asan/bin/kudu with pid 26174
2026-09-29T01:06:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4173 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9569 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (397728 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST